MVSFORUMS.com

[Deaf Smith]

Guys,

We have a huge old payroll system that has been replaced with a server. BUT, for bringing up old data on employees we would like to get the cobol/cics system off and on a server.

Anyone know about XML and what it takes to convert to that?

Thanks.
_________________
Deaf

[papadi]

What prompted the consideration of XML?

Do i understand correctly that the goal for this exercise is to get the data from the mainframe to the server database?

My teams have done hundreds of conversions/migrations from one platform to another (most often from a mainframe to *nix or Win-based targets, but some the other way as well). These resulted in several thousand tables on the targets. Some involved qsam files needed for online queries. Some moved vsam data and/or database data to a new target databases. Some were to provide the initial data to load a new datawarehouse. None of these used xml . . . fwiw.

One easy way to move data from the mainframe to another system is to create text files that are "loadable" directly on the target. Each target table would have a separate file created on the mainframe. These would be data-delimited text files and we usually use the tab character (x'05') as the delimiter. There will be no binary or packed-decimal fields in the downloaded file.

When stored on the target system, the file would be more understandable with a .csv extension. These are easily imported or loaded directly into the target databases.

If the goal is to get the historical data ported, any consideration of current transaction processing might not even need attention - that just depends on specifics.

Enterprise COBOL supports XML, but i'm not sure you need to go to that length to get what you need.
_________________
All the best,

di

[Deaf Smith]

Well the system we are thinking of, in converting to from the old legacy system, is it will have about 100 TB of data, if not more, in txt files that are delimited by pipes so we can load into either SQL or Oracle depending on if use use MS or Unix for the server. Space saving techniques like packed decimal for dates will be used as 100TB I'm not sure about (I presume DBs can span drives.)

Yes I figured out already to convert the VSAMS to tables and any VSAM with 'OCCURS depending on' into sub-tables and then relate them all by the main key used in the VSAMS. I can use VIEWS to limit the scope of searches to.

I am just not sure how fast (or slow) such a server will be. Many of the files have well over 9000 cylinders.

I have some ASP.NET background as well as Linux and SQL (my home PCs use Ubunto and Fedora), but most of my working life has been MVS/COBOL/CICS/JES and all that relates to it.

The main questions I have is if the XML conversions of very large systems onto servers will end up with slow servers that won't make users happy! I want to do this right the first time.

Thanks.
_________________
Deaf

[dbzTHEdinosauer]

you use xml as a media to transfer/communicate between applications,
because the applications are written to use xml - soap, etc.....

you are just transfering data from one system to another.
to introduce xml as a wrapper of this data,
is not suggested.
_________________
Dick Brenholtz
American living in Varel, Germany

[Deaf Smith]

Ok.

Now we would simply use FTP and COBOL to create flat files to send to a server and there create tables with SQL or Oracle.

But could XML be then used to communicate with the server to display data for users. That way they can see the old data from the legacy and we can shut off that LPAR from the mainframe.

dbz,

Is the 'wrapper' you are referring to the XML to retrieve the data for users?

Thanks
_________________
Deaf

[papadi]

I believe the "wrapper" mentioned was to support the transfer - and this is not needed.

Also, both Oracle and SQL Server have user presentation tools that do not need xml.

If you offload the data (split so that one file does not have 100tb) and load the data into the target(s) you could then use the presentation tools native to that target to retrieve data for the users.

And then the mainframe components could be phased out.

I'd suggest doing some prototyping with moderate amounts of data and making sure the users are in agreement with the new presentation style(s). Often this causes adventures. . .
_________________
All the best,

di

[Deaf Smith]

Ok. XML good for transferring data but bad for end-user systems.

We were planning on COBOL creating the flat files with pipes to delimit the data then FTP to the sever and rebuild. Would XML be used on the server to facilitate the FTP and rebuild and then load into a DB with such as SQL Server tools?

I think some of the tables will be in the 10 to 20 TB range, others less. Pay stubs that go back to 1980s would take the 20TB range files.

thanks.
_________________
Deaf

[dbzTHEdinosauer]

Deaf,

1. direct vsam to db2 layout was a salesman's bs for selling db2
and convincing management that there was no need for vast programming changes.

direct vsam to db2 allows one to access rows (previously vsam records)
without using primary key (db2 or vsam). otherwise, you are wasting db2's potential when you do not redesign the data structures and relationships.

you can replace db2 with any rdms system. oracle, ms-sql, etc...

you say you will have a different software suit as a payroll application
on the server side.
or
did you port the cobol from mainframe to server?

thru rationalization, you can often reduce the total data size for a system.
vsam is not conducive to this kind of thing, most everything about an 'account' is contained in the vsam record. whereas, a properly designed rdms will remove repetitive fields (the values) and store these one time in another table, instead of being repeated for every vsam record.
the idea is to removed duplicated values from the main vsam record.
a good example is city and state. there are a finite number of states,
and cities within the states. a two or 3 byte code to replace the potentially 30 byte cities in each vsam record would save a lot of space.

otherwise,

why you are going to have a separate system for the history and another for the current? if they we combined in the same system, the software (presentation layer) would be the same.

why are you worried about size? that is the nice thing about an rdms vs a vsam system. vsam is indeed limited to number of records, size of file, etc...
an rdms is not so limited - 100TB is large, but not unheard of. Major rdms's handle this amount of data easily. (though back-up and restore obviously requires more time than a 100GB system). as with any rdms, just gotta buy enough iron.....

i would forget about xml. xml is used to define each field to an application that is designed to input many various input types. in your case, you know the definition of each field (due to position).

also, instead of cobol, i would use dfsort/syncsort to reformat the vsam to pipe delimited qsam. much quicker.... especially since you are not converting from one file design to another, just extracting data.
_________________
Dick Brenholtz
American living in Varel, Germany

[Deaf Smith]

Mr. Brenholtz,

Thank you for your response. It's more help than you think!

Now I once have converted a COBOL/VSAM/CICS system to Microsoft SQL and ASP.NET by myself using a ASP.NET book and free 'visual developer 2005'. But I know so little about servers themselves!

So they can go to 100TB or more. Good. And 40 or so users at once.

Yes when I made the DB I did get rid of redundancies making the master VSAM into several tables depending to take out the redundancies (no body told me, I just well figured that was the way it had to work.) Yes rationalization!

And XML is just to define fields and not for screen handling or IO. Ok, my boss asked me about that and I'm looking at a XML tutorial this weekend (yes teach myself.)

The reason for separate system is the payroll system had a current (as of now) system and the backups that go back in time have different format (it's an archive that is.)

Now while I am extracting data it will be to design a DB system that will fit all on a server and give the users just what info they will determine they need. But knowing users that will change AFTER I start moving data.

The current payroll system is a package we bought and transfered current data to.. It is now several years old. The OLD VSAM system in on a ZOS and we want to shut down the ZOS. So that is why we need all the old data going back may years to be on a server and accessible to users.

I am sure I can use COBOL to turn the VSAMS into a rational number of tables and make a RDMS. To me that part is not hard at all, but size of the tables worried me. But when you say 100TB is not unheard of that means it can be done. Yes backing up will be, er, slow! I'll have to look into just how that is done and the ways to speed it up.

I am just not sure what the server size/configuration should be for 40 users, 100 TB, and what software platform! I can learn most of it no matter if it's Unix based or MS.

You can say I've been brought from the COBOL programmer side to the database admin. side.

Thanks again.
_________________
Deaf

[dbzTHEdinosauer]

an important point. buy more iron.

but, you will find that unless you invest wisely in server hardware,
you will have problems achieving the 99.999 percent uptime
that z/OS machines can boast of.

If i was you, during your self-training period,
you do a lot of surfing of forums dedicated to rdms on servers.

http://www.dbforums.com/index.php is a place to start. (not only db2, oracle,my-sql, ms-sql)
there are many dba's that maintain membership there who are highly qualified
(and opinionated)
but nevertheless, very helpful.

I have little experience of servers, i am a big blue z/os guy.
i have only heard the horror stories of server down time.

members like papadi can advise you better on that end.

They can also advise you on this last point:
i mean nothing negative about your abilities, competence,
but even a small system (yours is small - only one or two applications)
requires some expert/experienced advise - something you have to pay for.
Payroll is a sensitive application.
(unless you want to spend a weekend at end of month
manually typing out paychecks)
Also, since porting payroll history means the end of the z/os
that means you already have something going on the servers.
it really is a new world.
make you company pay for some help on this.

by the way, hats off to you, for your success so far.
_________________
Dick Brenholtz
American living in Varel, Germany

[Deaf Smith]

Thanks.

I was THE payroll programmer for 10 years here! Integral was the system.

Started on TI-990 mini computers when first out of college doing system generations, installing packages, and writing Cobol and TI/FORM (their idea of a screen handler like CICS.). Then VSE on a 390. Migrated to MVS with TSO/ISPF back in, oh, 1989 or so. Been on many applications and packages like Global AP/AR/GL as well as Integral payroll and HBOC (a hospital package my Mckesson.)

But I also am an administrator on Report2Web now and yes the ZOS is going, so it's adapt for perish!

Thanks for the link. I'll be asking them lots of questions as well. And yes, hardware, 'excess is the way to success.) I figure a mirror system to flash update so if anything happens the second system will step in to keep it going.

Thanks again.
_________________
Deaf

[dbzTHEdinosauer]

Back to the xml.

other than your boss wanting to know about a buzzword,
why the desire for xml?

xml is used to envelop data for transfer between disparate systems/applications (different sites/platforms)
webservices that communicate with a mainframe
/
websphere for instance uses xml enveloping to receive data destined for cics
and the reverse (cics output back to the server)
and
alot of jboss stuff uses xml to drive data transfers/application switching on servers.

nothing against micro-focus cobol (fujitsu also) on a server.
But it brings a very heavy footprint in order to run.
and companies invest in the software and hardware to run cobol on a server
because it is cheaper than replacing competent cobol programers with java/c types.

if you are going to be using xml,
learn to use it with the programming language that will be used on the platform.

using xml to provide an input stream
of the archived data to the new payroll system
i can see.

i understand why you would want your archived history
to be separate from your current data,
but i have reservations about using two different applications (current and archived).

i am not really trying to nix any ideas,
just make you realize that there is a lot planning/design/work involved.

again good luck
_________________
Dick Brenholtz
American living in Varel, Germany

[semigeezer]

Sorry - I didn't follow this thread closely - maybe this was covered... but one consideration... if you have 100TB of raw data, then consider that XML tags, white space and possibly attributes and structuring tags will increase the amount of storage used dramatically. If you have a 4 byte binary field x'98765432', then representing that in XML involves adding tags, and often data or format conversion. A simple case of converting a structure to XML and doing conversion might result in:
<outerstructure><somefield>2557891634</somefield></outerstructure>
so my 4 byte field is now represented with 66 bytes. If this ratio were to hold (admittedly it is a little excessive), that would turn your 100TB of data into 1.65PB (Petabytes) of XML. That's a lot to transmit and alot to store (though I heard a podcast today that said Walmart collects that much info from customers and transactions every hour).
_________________
New members are encouraged to read the How To Ask Questions The Smart Way FAQ at http://www.catb.org/~esr/faqs/smart-questions.html.