MVSFORUMS.com Forum Index MVSFORUMS.com
A Community of and for MVS Professionals
 
 FAQFAQ   SearchSearch   Quick Manuals   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Merging two Files - Removing Duplicates
Goto page Previous  1, 2
 
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> Utilities
View previous topic :: View next topic  
Author Message
singhnarender79
Beginner


Joined: 24 Dec 2002
Posts: 32
Topics: 6
Location: U.K

PostPosted: Mon May 23, 2005 11:21 am    Post subject: Reply with quote

Yes, you have got it right, I want the output records to be ordered according to the key.
_________________
"Hold fast to dreams, for if dreams die, life is a broken winged bird that cannot fly."

-- Langston Hughes
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose

PostPosted: Mon May 23, 2005 1:04 pm    Post subject: Reply with quote

Naren,

Here's a DFSORT/ICETOOL job that will do what you asked for. You'll need DFSORT's Dec, 2004 PTF to use the new IFTHEN, OVERLAY, KEEPBASE and VLENOVLY functions.

Code:

//S1    EXEC  PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN1 DD DSN=... input file1 (VB/90)
//IN2 DD DSN=... input file2 (VB/90)
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(MOD,PASS)
//T2 DD DSN=&&T2,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//T3 DD DSN=&&T3,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(MOD,PASS)
//OUT DD DSN=... output file (VB/90)
//TOOLIN DD *
  COPY FROM(IN1) TO(T1) USING(CTL1)
  COPY FROM(IN2) TO(T1) USING(CTL2)
  COPY FROM(T1) TO(T2) USING(CTL3)
  SELECT FROM(T1) TO(T3) -
    ON(51,1,PD) ON(52,2,CH) ON(54,2,PD) ON(56,3,PD) -
    FIRSTDUP USING(CTL4)
  SPLICE FROM(T2) TO(T3) ON(7,8,ZD) VLENOVLY KEEPBASE-
    WITHALL WITH(31,86)
  SPLICE FROM(T3) TO(OUT) -
    ON(23,1,PD) ON(24,2,CH) ON(26,2,PD) ON(28,3,PD) -
    VLENOVLY KEEPBASE-
    WITHALL WITH(5,1) WITH(31,86) USING(CTL5)
/*
//CTL1CNTL DD *
  INREC BUILD=(1,4,5:C'11',31:5)
/*
//CTL2CNTL DD *
  INREC BUILD=(1,4,5:C'22',31:5)
/*
//CTL3CNTL DD *
  INREC IFTHEN=(WHEN=INIT,OVERLAY=(7:SEQNUM,8,ZD)),
        IFTHEN=(WHEN=(31,1,BI,EQ,X'01'),
                OVERLAY=(7:SEQNUM,8,ZD,23:51,8)),
        IFTHEN=(WHEN=NONE,
                OVERLAY=(15:SEQNUM,8,ZD,
                         7:7,8,ZD,SUB,15,8,ZD,M11,LENGTH=8))
/*
//CTL4CNTL DD *
  INCLUDE COND=(31,1,BI,EQ,X'01')
  OUTFIL FNAMES=T3,OVERLAY=(5:C'XX',23:51,8)
/*
//CTL5CNTL DD *
  OUTFIL FNAMES=OUT,INCLUDE=(5,2,SS,EQ,C'2X,11,22'),
    OUTREC=(1,4,5:31)
/*

_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
singhnarender79
Beginner


Joined: 24 Dec 2002
Posts: 32
Topics: 6
Location: U.K

PostPosted: Tue May 24, 2005 3:21 pm    Post subject: Reply with quote

Hi Frank,

I tried to analyse the code you had send across as I was not able to execute it on our system (The PTF is getting installed on 29 May).

I took the following i/p files

File 1
X'01' Pol1
X'02'
X'02'
X'01' Pol2
X'02'
X'02'

File 2
X'01' Pol2
X'02'
X'02'
X'01' Pol4
X'02'
X'02'


Our T3 file looks like this...
Code:

Col 5   Col 7   Col 23   Col 31   Col 51

XX       Pol1   X'01'   Pol1
XX       Pol2   X'01'   Pol2 frm File1
XX       Pol3   X'01'   Pol4    
11   1   Pol1   X'01'   Pol1
11   1   Pol1   X'02'   Pol1
11   1   Pol1   X'02'   Pol1
11   2   Pol2   X'01'   Pol2 frm File1
11   2   Pol2   X'02'   Pol2 frm File1
11   2   Pol2   X'02'   Pol2 frm File1
22   3   Pol2   X'01'   Pol2 frm File2
22   3   Pol2   X'02'   Pol2 frm File2
22   3   Pol2   X'02'   Pol2 frm File2
22   4   Pol4   X'01'   Pol4
22   4   Pol4   X'02'   Pol4
22   4   Pol4   X'02'   Pol4

On using the SPLICE from T3 to OUT, I am not getting the expected result. Cols 5 & 6 in OUT contain XX, 1X AND 2X. In CTL5 you have mentioned 2X, 11 and 22. It would be extremely helpful if you can verify my T3 file and let me know if I have done some mistake in my analysis.

Thanks,
Naren
_________________
"Hold fast to dreams, for if dreams die, life is a broken winged bird that cannot fly."

-- Langston Hughes
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose

PostPosted: Tue May 24, 2005 5:29 pm    Post subject: Reply with quote

Naren,

What you're saying makes no sense to me. You say "On using the SPLICE from T3 to OUT, I am not getting the expected result". But you also say you don't have the Dec, 2004 PTF installed yet. So how can you be running the SPLICE operator when it uses KEEPBASE and VLENOVLY which you can't use yet because you don't have the PTF?

You told me in an earlier post that your "key" starts in position 25, is 8 bytes long and consists of these four fields:

MP-POL-DEPT PIC S9 COMP-3.
MP-POL-LET PIC XX.
MP-POL-YEAR PIC S9(3) COMP-3.
MP-POL-SERIAL PIC S9(5) COMP-3.

That's what I set up the job to use for the keys.

I don't know what your input of:

X'01' Pol1
...

represents, but it doesn't look like the keys you described and for which the job is set up.

Also, T3 is a MOD data set used as the output for TWO operators. I have no idea which operator your T3 records resulted from. Again, if it's the output of the SPLICE operator, how did you use KEEPBASE and VLENOVLY when you don't have the PTF installed yet?

I don't really understand what it is you're doing or what you want me to analyze. I tested the job using input I created based on what you told me about the input. The job gave the expected output. If you want me to try the job with different input, you're going to have to supply that input in its exact form to me.
_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose

PostPosted: Tue May 24, 2005 5:38 pm    Post subject: Reply with quote

Naren,

BTW, I assumed that your 8-byte key starts in position 25 counting the 4-byte RDW. That is, it looks like this:

Code:

1   5      25
|RDW|......|key|
|-4-|--20--|-8-|


If it really starts in position 29 when counting the RDW and looks like this:

Code:

1   5      29
|RDW|......|key|
|-4-|--24--|-8-|


then I'd have to adjust the job for that difference.
_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
singhnarender79
Beginner


Joined: 24 Dec 2002
Posts: 32
Topics: 6
Location: U.K

PostPosted: Wed May 25, 2005 3:41 am    Post subject: Reply with quote

Frank,

I was trying to understand the code which you had given me. Since you have used some new functions, I had looked for them in the Application Programming Guide and tried to understand what exactly they are doing in code you gave me. And since the PTF is not yet installed, I tried to figure out the output manually (On Paper). (i.e by writing down some input records, then figuring out what each step was doing and writing the output from each step)

The key which i mentioned is right and it starts from 25 (excluding the RDW, but that we had figured out) i.e

P-POL-DEPT PIC S9 COMP-3.
MP-POL-LET PIC XX.
MP-POL-YEAR PIC S9(3) COMP-3.
MP-POL-SERIAL PIC S9(5) COMP-3.

What I mentioned above was the first two bytes that comes before the key

X'0101'SMOPHRT
X'0202'SMOPHNAM
X'0302'SMOPHADD
X'0302'SMOPHADD
X'0402'SMOPHOCC

and I think u have also used that (Correct me if i am wrong)

Quote:
//CTL3CNTL DD *
INREC IFTHEN=(WHEN=INIT,OVERLAY=(7:SEQNUM,8,ZD)),
IFTHEN=(WHEN=(31,1,BI,EQ,X'01'),
OVERLAY=(7:SEQNUM,8,ZD,23:51,8 )),
IFTHEN=(WHEN=NONE,
OVERLAY=(15:SEQNUM,8,ZD,
7:7,8,ZD,SUB,15,8,ZD,M11,LENGTH=8 ))
/*
//CTL4CNTL DD *
INCLUDE COND=(31,1,BI,EQ,X'01')
OUTFIL FNAMES=T3,OVERLAY=(5:C'XX',23:51,8 )


The output which I had mentioned is the resultant of the both the operators SELECT and SPLICE.

But as you say you got the right ouput (Unique records from File1 + Unique Records from File2 + Records common in both, but taken from File2), I might have done some mistake while analysing the code manually. I will try the code when the PTF gets installed on 29th.

Thanks for your help,
Naren Smile
_________________
"Hold fast to dreams, for if dreams die, life is a broken winged bird that cannot fly."

-- Langston Hughes
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose

PostPosted: Wed May 25, 2005 10:12 am    Post subject: Reply with quote

Yes, I used the X'01' in position 5 as the indicator that a new group was starting.
_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
singhnarender79
Beginner


Joined: 24 Dec 2002
Posts: 32
Topics: 6
Location: U.K

PostPosted: Tue May 31, 2005 5:11 am    Post subject: Reply with quote

Frank,

We have got the new PTF installed and I have executed the code you gave me. I am very happy to say that it's working like a charm.

and also i found out, where i had done a mistake while doing it manually.

Thanks for all the help.

Cheers,
Naren
_________________
"Hold fast to dreams, for if dreams die, life is a broken winged bird that cannot fly."

-- Langston Hughes
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose

PostPosted: Tue May 31, 2005 10:07 am    Post subject: Reply with quote

Thanks for the feedback. I'm glad I could help.
_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
singhnarender79
Beginner


Joined: 24 Dec 2002
Posts: 32
Topics: 6
Location: U.K

PostPosted: Thu Jun 02, 2005 11:17 am    Post subject: Reply with quote

Frank,

There is one more interesting development that has come in my requirement. I have a workaround for that, but i was just wondering if it can be possible using DFSORT. So, if u can help us out it would be great.

The orinal req. is same but with one more addition. The addition is that when we are merging the 2 files, giving priority to records from file 2(which are also present in file 1), we need to extract some info (a date field) from the record in File1 and overlay it on the same record from file 2, and put it in the output file. If this is confusing, here is an example which explains it....

Pol means Policy field.

Input File 1

..SMOPHRT Pol1
..SMOPHNAM date1
..SMOPHADD
..SMOPHRT Pol2
..SMOPHNAM date2
..SMOPHADD

Input File 2
..SMOPHRT Pol3
..SMOPHNAM date3
..SMOPHADD
..SMOPHRT Pol2
..SMOPHNAM date4
..SMOPHADD

Output File

..SMOPHRT Pol1
..SMOPHNAM date1
..SMOPHADD
..SMOPHRT Pol2
..SMOPHNAM date2
..SMOPHADD
..SMOPHRT Pol3
..SMOPHNAM date3
..SMOPHADD

As you can see above Pol2 was common in both but taken from File2, and only the date has to be taken from Pol2 record from file1. So in the Output file, date4 was replaced by date2 for Pol2 record. As you can see above, date is not present in the ROOT record but in the next record.

The date field is defined as PIC S9(4) COMP and the Position is 56 (excluding RDW).

Thanks,
Naren
_________________
"Hold fast to dreams, for if dreams die, life is a broken winged bird that cannot fly."

-- Langston Hughes
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose

PostPosted: Thu Jun 02, 2005 11:54 am    Post subject: Reply with quote

So you want the date from the X'02' record in file1 for matching groups? Can it be the entire X'02' record in file1, or does it have to be just the date from the X'02' record in file1 and the rest of the fields from the X'02' record in file2?
_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
singhnarender79
Beginner


Joined: 24 Dec 2002
Posts: 32
Topics: 6
Location: U.K

PostPosted: Thu Jun 02, 2005 12:31 pm    Post subject: Reply with quote

It has to be just the date from the X'02' record in file1 and the rest of the fields from the X'02' record in file2 for the matching group only.
_________________
"Hold fast to dreams, for if dreams die, life is a broken winged bird that cannot fly."

-- Langston Hughes
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose

PostPosted: Thu Jun 02, 2005 1:15 pm    Post subject: Reply with quote

Naren,

I think I've come up with a way to do it, but it will take me a little time to work it out. I'll post the solution later today.
_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Frank Yaeger
Sort Forum Moderator
Sort Forum Moderator


Joined: 02 Dec 2002
Posts: 1618
Topics: 31
Location: San Jose

PostPosted: Thu Jun 02, 2005 2:35 pm    Post subject: Reply with quote

Naren,

Here's the revised DFSORT/ICETOOL job to handle the new requirement.

Code:

//S1    EXEC  PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN1 DD DSN=... input file1 (VB/90)
//IN2 DD DSN=... input file2 (VB/90)
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(MOD,PASS)
//T2 DD DSN=&&T2,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//T3 DD DSN=&&T3,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(MOD,PASS)
//T4 DD DSN=&&T4,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(MOD,PASS)
//OUT DD DSN=... output file (VB/90)
//TOOLIN DD *
  COPY FROM(IN1) TO(T1) USING(CTL1)
  COPY FROM(IN2) TO(T1) USING(CTL2)
  COPY FROM(T1) TO(T2) USING(CTL3)
  SELECT FROM(T1) TO(T3) -
    ON(51,1,PD) ON(52,2,CH) ON(54,2,PD) ON(56,3,PD) -
    FIRSTDUP USING(CTL4)
  SPLICE FROM(T2) TO(T3) ON(7,8,ZD) VLENOVLY KEEPBASE-
    WITHALL WITH(31,86)
  SPLICE FROM(T3) TO(T4) -
    ON(23,1,PD) ON(24,2,CH) ON(26,2,PD) ON(28,3,PD) ON(31,1,BI) -
    VLENOVLY KEEPNODUPS-
    WITH(5,2) WITH(31,55) WITH(88,29) USING(CTL5)
  COPY FROM(T3) TO(T4) USING(CTL6)
  SORT FROM(T4) TO(T2) USING(CTL7)
  SPLICE FROM(T2) TO(OUT) -
    ON(23,1,PD) ON(24,2,CH) ON(26,2,PD) ON(28,3,PD) -
    VLENOVLY KEEPBASE-
    WITHALL WITH(5,1) WITH(31,86) USING(CTL8)
/*
//CTL1CNTL DD *
  INREC BUILD=(1,4,5:C'11',31:5)
/*
//CTL2CNTL DD *
  INREC BUILD=(1,4,5:C'22',31:5)
/*
//CTL3CNTL DD *
  INREC IFTHEN=(WHEN=INIT,OVERLAY=(7:SEQNUM,8,ZD)),
        IFTHEN=(WHEN=(31,1,BI,EQ,X'01'),
                OVERLAY=(7:SEQNUM,8,ZD,23:51,8)),
        IFTHEN=(WHEN=NONE,
                OVERLAY=(15:SEQNUM,8,ZD,
                         7:7,8,ZD,SUB,15,8,ZD,M11,LENGTH=8))
/*
//CTL4CNTL DD *
  INCLUDE COND=(31,1,BI,EQ,X'01')
  OUTFIL FNAMES=T3,OVERLAY=(5:C'XX',23:51,8)
/*
//CTL5CNTL DD *
  INCLUDE COND=(31,1,BI,EQ,X'02')
/*
//CTL6CNTL DD *
  INCLUDE COND=(31,1,BI,NE,X'02')
/*
//CTL7CNTL DD *
  SORT FIELDS=(23,1,PD,A,24,2,CH,A,26,2,PD,A,
    28,3,PD,A,31,1,BI,A)
/*
//CTL8CNTL DD *
  OUTFIL FNAMES=OUT,INCLUDE=(5,2,SS,EQ,C'2X,11,22'),
    OUTREC=(1,4,5:31)
/*

_________________
Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Back to top
View user's profile Send private message Send e-mail Visit poster's website
singhnarender79
Beginner


Joined: 24 Dec 2002
Posts: 32
Topics: 6
Location: U.K

PostPosted: Mon Jun 06, 2005 6:28 am    Post subject: Reply with quote

Hi Frank,

Thanks again for all the help. It is also working perfectly, but had to modify it slightly to get the records in the same order as in the input file. (I used the sequence number in sort rather then the Binary values).

Cheers,
Naren
_________________
"Hold fast to dreams, for if dreams die, life is a broken winged bird that cannot fly."

-- Langston Hughes
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> Utilities All times are GMT - 5 Hours
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


MVSFORUMS
Powered by phpBB © 2001, 2005 phpBB Group