View previous topic :: View next topic |
Author |
Message |
moyeenkhan Beginner
Joined: 04 Dec 2002 Posts: 64 Topics: 21
|
Posted: Wed Feb 16, 2011 12:04 pm Post subject: What is the best way to sort 2 Million records |
|
|
What is the best and efficient way to sort 2 million records. Do you think MAXSORT is the way to go? |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
kolusu Site Admin
![Site Admin Site Admin](images/ranks/adminpic.jpg)
Joined: 26 Nov 2002 Posts: 12369 Topics: 75 Location: San Jose
|
Posted: Wed Feb 16, 2011 12:09 pm Post subject: |
|
|
moyeenkhan,
Depends on lots of factors.
1. LRECL
2. RECFM
3. Length of the key to be sorted.
4. Format of the key to be sorted.
5. Availability of work datasets/Main storage.... _________________ Kolusu
www.linkedin.com/in/kolusu |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
papadi Supermod
Joined: 20 Oct 2009 Posts: 594 Topics: 1
|
Posted: Wed Feb 16, 2011 1:42 pm Post subject: |
|
|
Several times a month i sort (well, not personally<g> - this is scheduled) between 8 and 10 million records that are more than 1k each (the whole record is used not a subset - see below). This typically runs in a few minutes. . .
One way to speed up a sort is if the sorted output does not need all of the data in the input records. A subset containing the sort-key(s) and needed data may run exponentially faster than sorting the whole set of data. Not only will this run faster but there will be less need for sortwork and output file dasd (which sometmes encounters space allocation problems).
Lots of times the entire set of data is sorted because it takes so little work to simply "sort it all". The process i mentioned earlier sorts the full data because the data as received is not in the correct sequence for most of the processing that uses this data. _________________ All the best,
di |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
Frank Yaeger Sort Forum Moderator
![Sort Forum Moderator Sort Forum Moderator](images/ranks/profrat.jpg)
Joined: 02 Dec 2002 Posts: 1618 Topics: 31 Location: San Jose
|
Posted: Wed Feb 16, 2011 1:44 pm Post subject: |
|
|
moyeenkhan,
Which sort product are you using?
Have you tried sorting your 2 million records with a "normal" sort and run into a "problem"?
You really need to give more information before anyone can help you. _________________ Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
moyeenkhan Beginner
Joined: 04 Dec 2002 Posts: 64 Topics: 21
|
Posted: Wed Feb 16, 2011 2:26 pm Post subject: |
|
|
Many thanks for your time and response.
Here are moe details:
We have Syncsort
The input file is VB - 7508 Bytes
Sort Key details are
OPTION Y2PAST=1951
SORT FIELDS=(45,5,CH,A,94,2,CH,A,132,1,Y2D,A,133,2,PD,A)
RECORD TYPE=V,LENGTH=9674
Currently using MAXSORT |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
Frank Yaeger Sort Forum Moderator
![Sort Forum Moderator Sort Forum Moderator](images/ranks/profrat.jpg)
Joined: 02 Dec 2002 Posts: 1618 Topics: 31 Location: San Jose
|
Posted: Wed Feb 16, 2011 4:14 pm Post subject: |
|
|
I'm a DFSORT guy, not a Syncsort guy, but FWIW, I'd be surprised if MAXSORT was really needed for this. I would think that Syncsort could do this without MAXSORT. _________________ Frank Yaeger - DFSORT Development Team (IBM)
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
DFSORT is on the Web at:
www.ibm.com/storage/dfsort |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
moyeenkhan Beginner
Joined: 04 Dec 2002 Posts: 64 Topics: 21
|
Posted: Wed Feb 16, 2011 5:19 pm Post subject: |
|
|
Thanks Frank. I also think I do not need MAXSORT. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
Bill Dennis Advanced
![](images/avatars/gallery/cartoons/WB60.gif)
Joined: 03 Dec 2002 Posts: 579 Topics: 1 Location: Iowa, USA
|
Posted: Wed Feb 16, 2011 10:59 pm Post subject: |
|
|
moyeenkhan,
MAXSORT is not for efficiency, it is for constrained SORTWORK space. The intermediate sort and merge phases require more resources. It is best reserved for sorting enormous files when the required work space cannot be allocated. _________________ Regards,
Bill Dennis
Disclaimer: My comments on this foorum are my own and do not represent the opinions or suggestions of any other person or business entity. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
moyeenkhan Beginner
Joined: 04 Dec 2002 Posts: 64 Topics: 21
|
Posted: Thu Feb 17, 2011 10:22 am Post subject: |
|
|
Thanks Bill. |
|
Back to top |
|
![](templates/subSilver/images/spacer.gif) |
|
|