View previous topic :: View next topic |
Author |
Message |
infoman123 Beginner
Joined: 02 Nov 2004 Posts: 57 Topics: 20
|
Posted: Wed Jun 09, 2010 5:55 am Post subject: Date routine performance optimization |
|
|
Hi All,
In our system, there are lot of places where lot of cobol logic performing Julian to Greg, Greg to Jul , elapsed days between 2 julian dates, future & back date from a Julian date is performed.
Currently, these routines are performed for each record in the system. We are trying to look for a tuning opportunity here. In such a way that suppose for 1st recod, if these date checks are performed store it somewhere and if any of the later records are with same date use the already calculated value earlier for 1st record and avoid the date routine execution.
For eg
Record 1 date is 2010159 and it has to perform jul to greg, and future date after 1 month, back date before 1 month etc
Record 2 date is 2010160 and it has to perform jul to greg, and future date after 1 month, back date before 1 month etc
Record 3 date is 2010161 and it has to perform jul to greg, and future date after 1 month, back date before 1 month etc
--
--
--
Record 10 date is 2010159 and it has to perform jul to greg, and future date after 1 month, back date before 1 month etc
Record 11 date is 2010160 and it has to perform jul to greg, and future date after 1 month, back date before 1 month etc
What we are looking for is already the calculation for 2010159, 2010160 has performed for record 1 & 2, so these value has to store somewhere and should be used for Record 10 & 11 without recalculating it.
If the date routines are already performed for earlier records, the idea is to re use this for later records without again calculating it.
Can someone help, how can we achieve this with cobol. This will defenitely help us to save our batch window.
Regards |
|
Back to top |
|
|
dbzTHEdinosauer Supermod
Joined: 20 Oct 2006 Posts: 1411 Topics: 26 Location: germany
|
Posted: Wed Jun 09, 2010 6:22 am Post subject: |
|
|
well, don't know if you are running IMS, DB2 or vsam.
one thing to consider is that if you attempt, during the run, to 'store' the calculations,
the extra cost of 'storing' the data has to be added to your run time.
then you have the additional overhead of reorg, etc....
there are only 366 potential days in a year. for a 10 year-time frame, this would be 3660 entries someplace (where ever you are going to store).
you have only shown us one calculation. how many types of calculations could be performed on one date? only adding 1 month, only subtracting 1 month.
if it is that simple, I would write a one-off program to populate whatever storage facility (ims, db2, vsam) that you employ. This program would have to take into consideration holidays, etc... But the program would only be run occasionally (yearly to extend for new, and to delete those dates that are too old)
each record/row would contain the julian date, the greg, and the calculated dates.
a simple read would replace the calculation routines.
properly buffered it could possibly be quicker that recalculation.
but again, keep in mind you are replacing potential CPU-boundness with I/O-boundness _________________ Dick Brenholtz
American living in Varel, Germany |
|
Back to top |
|
|
dbzTHEdinosauer Supermod
Joined: 20 Oct 2006 Posts: 1411 Topics: 26 Location: germany
|
Posted: Wed Jun 09, 2010 8:00 am Post subject: |
|
|
still don't know what you are doing, but if the date calculations are so intensive,
sort the file on date and expand the records to include the calculated dates.
that way you could off-load all the date calculation to one program,
since the file is sorted on date,
you could maintain the last calculated and skip the calculations if dates are same. _________________ Dick Brenholtz
American living in Varel, Germany |
|
Back to top |
|
|
Dibakar Advanced
Joined: 02 Dec 2002 Posts: 700 Topics: 63 Location: USA
|
Posted: Wed Jun 09, 2010 12:59 pm Post subject: |
|
|
I beleive logic for Greg to Julian conversion and looking ahead and back a month would be simple. So you can as well copy the logic of the called program in the caller to save time on calls.
Also check your date conversion utility to see if it can be optimised, it may be doing more calls and conversions at backend then you may need. _________________ Regards,
Diba |
|
Back to top |
|
|
infoman123 Beginner
Joined: 02 Nov 2004 Posts: 57 Topics: 20
|
Posted: Thu Jun 10, 2010 1:08 am Post subject: |
|
|
Thanks all.
Apologies if my explanation are not clear
Suppose these below are the fields in one record of a file. I am trying to explain it in a simple way but the logic of our program is quite complex to explain here.
Record1 |
|
Back to top |
|
|
dbzTHEdinosauer Supermod
Joined: 20 Oct 2006 Posts: 1411 Topics: 26 Location: germany
|
Posted: Thu Jun 10, 2010 3:30 am Post subject: |
|
|
you are going to expend resources 'saving calculation results'.
why not sort the file in date order? _________________ Dick Brenholtz
American living in Varel, Germany |
|
Back to top |
|
|
prino Banned
Joined: 01 Feb 2007 Posts: 45 Topics: 5 Location: Oostende
|
Posted: Thu Jun 10, 2010 5:30 am Post subject: |
|
|
The following code converts a date to a Julian Day Number, all arithmetic is integer:
Code: | {***********************************************************************
* CJ: *
* *
* Convert a date to its Julian Day Number *
* *
* Original code used: *
* *
* var cj_y: double; *
* var cj_c: double; *
* *
* cj_y:= dyear + (dmonth - 2.85) / 12; *
* cj_c:= 0.75 * trunc(cj_y / 100); *
* jdn:= trunc( *
* trunc( *
* trunc(367 * cj_y) - 1.75 * trunc(cj_y) + dday) - cj_c) + *
* 1721115; *
* *
* Reworked code to use only integer arithmetic: *
* *
* var c : longint; *
* var iy: longint; *
* var ia: longint; *
* var ib: longint; *
* var it: longint; *
* *
* begin *
* iy:= dyear * 88080 + dmonth * 7340 - 20919) div 240; *
* it:= dyear * 1200 + dmonth * 100 - 285; *
* ia:= it div 1200; *
* ib:= (iy * 4 - ia * 7 + dday * 4) div 4; *
* c := 3 * (it div 120000); *
* jdn:= (ib * 4 - c) div 4 + 1721115; *
***********************************************************************} |
and this does the inverse:
Code: | ************************************************************************
* JC: *
* *
* Convert a Julian Day Number to a date *
* *
* Original code used: *
* *
* var n: double; *
* var c: double; *
* *
* n:= 1.0 * jdn - 1721119.2; *
* c:= trunc(n / 36524.25); *
* *
* if jdn < 2299161 then *
* n:= n + 2 *
* else *
* n:= n + c - trunc(c / 4); *
* *
* yyyy:= trunc(n / 365.25); *
* n := n - trunc(365.25 * yyyy) - 0.3; *
* mm := trunc(n / 30.6); *
* dd := trunc(n - 30.6 * mm + 1); *
* *
* if mm > 9 then *
* begin *
* dec(mm, 9); *
* inc(yyyy); *
* end *
* else *
* inc(mm, 3); *
* *
* Reworked code to use only integer arithmetic: *
* *
* var n: longint; *
* var c: longint; *
* *
* n:= 10 * jdn - 17211192; *
* c:= (n * 10) div 3652425; *
* *
* if jdn < 2299161 then *
* n:= n + 20 *
* else *
* n:= n + (c - c div 4) * 10; *
* *
* yyyy:= (n * 10) div 36525; *
* n := n - 3650 * longint(yyyy) - 10 * (yyyy div 4) - 3; *
* mm := n div 306; *
* dd := (n - 306 * mm + 10) div 10; *
* *
* if mm > 9 then *
* begin *
* dec(mm, 9); *
* inc(yyyy); *
* end *
* else *
* inc(mm, 3); *
***********************************************************************} |
Both routines are fast but if you were to code them in assembler, replacing the divides by multiplies and shifts, they would be even faster. CJ graciously handles overflowed dates, ie 2010-01-32 is treated as 2010-02-01. Code is Pascal. |
|
Back to top |
|
|
Dibakar Advanced
Joined: 02 Dec 2002 Posts: 700 Topics: 63 Location: USA
|
Posted: Thu Jun 10, 2010 11:35 am Post subject: |
|
|
Quote: |
Suppose In the program, for each record,
the elapsed days between todays date and process date are calculated to perform a special processing logic. Like if its less than 15 do process 1 and if its > 15 do process 2 etc
if membership date = todays date, the calculate new membership date 1 year future date.
If expire date = todays date, the calculate new expiry date 1 year future date or future 1 month extension date based on some other condition.
If finance date = todays date, calculate the back date of finance due date by 1 month and perform some logic.
|
Looks like you need only few dates - today, today -15, today + 1 year, today + 1 month, today -1 month. So why not store these 5 dates before processing the records. Then for each record you can compare the date fields with with these dates instead of calling date routines. _________________ Regards,
Diba |
|
Back to top |
|
|
infoman123 Beginner
Joined: 02 Nov 2004 Posts: 57 Topics: 20
|
Posted: Fri Jun 11, 2010 3:58 am Post subject: |
|
|
Thanks Prino..
For converting JD to greg and viceversa..dont need this complex logic... basically if we hard code all the equivalent dates in working storage for leap and non leap year.. thts more faster.. our program is coded like that...
Dibakar... as i explained these" - today, today -15, today + 1 year, today + 1 month, today -1 month" but actual program logic is more complex than this. all these date calculation in our program is based on some parameters defined.
basically what i'm looking is something like this
suppose i want find elapsed days between 2 dates date 1 => 2010159 & date 2 => 2010156, then if its caluclated already for rec 1, for for later records if the elapsed days required for date 1 & date 1, the ws-elapsed-days (date1,date2) should give result 3 which is alredy calculated. |
|
Back to top |
|
|
Dibakar Advanced
Joined: 02 Dec 2002 Posts: 700 Topics: 63 Location: USA
|
Posted: Mon Jun 14, 2010 11:31 am Post subject: |
|
|
informan123,
I think you need to answer two questions from Dick -
Why are you not pre sorting on date
How many date calculations are you expecting to save in memory
Also what is the date window of your calculations. From your questions it looks like the dates are falling in a short window, maybe couple of months from current date, thats why you expect lots of repeatation. If so specify the window for all the date fields that may occur in your calculation. _________________ Regards,
Diba |
|
Back to top |
|
|
jsharon1248 Intermediate
Joined: 08 Aug 2007 Posts: 291 Topics: 2 Location: Chicago
|
Posted: Mon Jun 14, 2010 1:33 pm Post subject: |
|
|
What leads you to this conclusion?
Quote: | This will defenitely help us to save our batch window. |
I have my doubts. What makes you believe that converting dates is taking so much time? What analysis have you done that makes you believe that eliminating the date conversions is going to save elapsed time? |
|
Back to top |
|
|
infoman123 Beginner
Joined: 02 Nov 2004 Posts: 57 Topics: 20
|
Posted: Thu Jun 17, 2010 7:22 am Post subject: |
|
|
jsharon1248 wrote: | What leads you to this conclusion?
I have my doubts. What makes you believe that converting dates is taking so much time? What analysis have you done that makes you believe that eliminating the date conversions is going to save elapsed time? |
This is made based on looking the actual data in system. Daily there are lot of financial transactions happened are added to system. 80% of the input file from customer reps are on same day. 10% in same date in a week and 5 %in same month and 5% same year. But the logics are generic enough to handle any dates.
so based on this assumption only,if any date calculation is made same somewhere like in a scratch pad and use it as and when needed. If its not calculated perform the normal routine..
Regards |
|
Back to top |
|
|
infoman123 Beginner
Joined: 02 Nov 2004 Posts: 57 Topics: 20
|
Posted: Thu Jun 17, 2010 7:25 am Post subject: |
|
|
Dibakar wrote: |
I think you need to answer two questions from Dick -
Why are you not pre sorting on date
How many date calculations are you expecting to save in memory
Also what is the date window of your calculations. From your questions it looks like the dates are falling in a short window, maybe couple of months from current date, thats why you expect lots of repeatation. If so specify the window for all the date fields that may occur in your calculation. |
jsharon1248 wrote: | What leads you to this conclusion?
I have my doubts. What makes you believe that converting dates is taking so much time? What analysis have you done that makes you believe that eliminating the date conversions is going to save elapsed time? |
We cannot pre sort based on dates, then the whole processing logic needs to be changed.. the file is stored with a key of id number. |
|
Back to top |
|
|
jsharon1248 Intermediate
Joined: 08 Aug 2007 Posts: 291 Topics: 2 Location: Chicago
|
Posted: Thu Jun 17, 2010 7:57 am Post subject: |
|
|
You really haven't answered the question. Look at it this way. If you eliminated the date conversions all together, how much elapsed time would you save? Now, having said that, if 80% of the transactions are using the same date, why not add a snippet of code to bypass the date conversions if the dates on 2 consecutive records are the same. The converted values will already be in working storage. In your testing phase, add counters to determine how many date conversion are bypassed. Now you're in a position to run a parallel with the prod version to determine how much time you're really saving. Something like this:
Code: | IF INPUT-DATE NOT = WS-DATE
PERFORM DATE-CONVERSION-PARA
END-IF |
|
|
Back to top |
|
|
Dibakar Advanced
Joined: 02 Dec 2002 Posts: 700 Topics: 63 Location: USA
|
Posted: Thu Jun 17, 2010 12:43 pm Post subject: |
|
|
I am trying to extend the solution propsed by jsharon1248.
infoman123 wrote: | 80% of the input file from customer reps are on same day. |
infoman123 wrote: | 10% in same date in a week |
I am guessing your program can determine what are these most occuring date and week are.
Create a table to save date calculations for this date and all the days of the most frequent week. Have the most frequent date as the first row.
Now 90% of calculations are saved without taking two much space. And scanning through them to find your date won't take too much time either.
You may extend this logic to save more date calculations that you think are frequent, put more frequent ones at top.
If you don't have logic to calculate most frequent dates then I beleive solution proposed by jsharon1248 is the only work around. _________________ Regards,
Diba |
|
Back to top |
|
|
|
|