GUIDELINE FOR MAKING DSE DATA CAPTURE TOOLS





1. EOD DATA CAPTURE TOOL (like DDD).

2. TRADE-HOUR DATA CAPTURE TOOL (like DSNAP).


We have 2 windows-based tools for capturing EOD-data for DSE (Dhaka stock exchange). These are DDU and DDD. Respected maxether bhai and abyss bhai created and distributed these for free.
And we have a trade-time data capture tool for windows, DSNAP. This one is made and gifted by abyss bhai for free.
We need tools that can work on Linux. Linux users are increasing very quickly.

I was talking about this matter with a friend (Sarim Khan) who is a good programmer and Linux expert. Unfortunately he is not familiar with stock market data. I had to explain the whole thing. Again unfortunately I am not a programmer, may be I could not explain properly. Sarim bhai assured me, he can make tools that will work on any operating system, Linux, windows, Mac, Solaris etc. He told, java based tools work on any os.

In the Linux-world, everything is free and open-source. Millions of big-heart people are contributing without taking any benefit.



Here is some email-conversations I have had with my programmer friend:

---------- ---------- -------------

we use charts for seeing stock market conditions.
we use amibroker for making charts. www.amibroker.com
international stock markets supply csv files to use in amibroker. but bangladeshi stock market gives dirty format data which is not workable. so we need to convert these dirty data.

2 types data we need. EOD or end of day. market remains open from 11am to 3pm. after 3 pm we make eod chart to see the whole day picture.
'till now data' is used for seeing the market condition during trade hour. this is not 'intraday data', though it is like intraday.

csv file must contain six parameters. date open high low close volume.
date is the date of that specific trading day.
open means opening price or the first price of that day.
high is the highest price of the day.
low is the lowest price of the day.
close is the closing price of the day, not last price.
volume is number of stocks traded in that day.


--------------- MAKING JAVA BASED DDD ----------------
-------------- EOD DATA
CAPTURE TOOL ------------

for eod data we use this link:
http://admin.dsebd.org/admin-real/mst.txt
or this link:
http://webnew.dsebd.org/admin-real/mst.txt
or this link:
http://adminnew.dsebd.org/admin-real/mst.txt

see, dse is giving 3 links for a same txt file. sometimes any one or
two remains unavailable. probably they will change their link again.
so, we need an option to put the link manually. it can be so: the tool
should try above three links one after another if fails it will try
the link which i will write manually. see the ddd, i can manually
change the sources, 4 options r there but now option for manually
writing address. ddd can not search one after another. it grabs data
only from the given single link. admin / web / admin new / web new. In
dse home page, www.dsebd.org or www.dse.com.bd , see a link at left
panel lower area, ''market statistics''. this is the source of mst
file. it comes 30 minutes after closure of trade.

again, for eod data, we need these: date, ticker name, open, high,
low, close, volume.

For intraday data, we need: date, time, ticker name, open, high, low, close, volume.

we can save data as 'per stock csv' or 'per day csv' . ddd creates
automatically: folders, csv files, new ticker file when a new stock
comes in market.

Also see: public, spot, block, odd-lot transactions (in mst file). we will add public and
spot. block market will not be added in our data. (odd lot will not be added too).
volume means, number of stocks traded on that day. not the value in taka.

Also see, dse general index will be added in our data. file/ticker
name is 00dsegen (zero zero dsegen). here volume is total value in taka here. not the
number of trades. volume will be divided by 100 automatically and kept
in our output data.

note: hi and lo of 00dsegen is not available in mst. we will grab it
from this link:
http://web.dsebd.org/admin-real/index-graph/companygraph.php?graph_id=gen

see, ''get from file'' and ''get from web''. From file means, from
saved mst file. no index update is possible here. because hi and low
of index is not available in mst file.

i do not use other options in ddd. these are SECTOR, EQUITY, SECTOR
DEF UPDATE, GENERATE SECTOR ARCHIVE. Please do not waste your time for
these options. unnecessary things. we can make these inside amibroker
automatically.



-------------- MAKING JAVA BASED DDD ----------------
-------------- EOD DATA
CAPTURE TOOL ----------------

Please open a csv file from ''per day csv folder'', and another from
''per stock csv folder''.
see the difference. also see the number of csv files inside these folders.

I believe, per-stock option should be discarded totally. this way we
can save our valuable time. this method has some problems too. I have
last 10 yrs' full database at per stock format. it is getting bigger
and bigger day by day. very difficult to send via email. per day is
small. if i fail to update at a specific day, i can ask a friend to
send it via email. he can send only one file of that day, only 15
kilobyte size.
I saved last 10 yrs data as per stock format up to 31-12-2009, kept it
for historical archive. Now saving only per day format since 1-1-2010.

I will request you to omit per stock format. it will save your time.

just see a csv file (from per day folder). you will understand easily,
how it is made. you r a talented programmer, as far i understood.

--------------------


Good news is, I found, ddd and dsnap both works without ms-office. I
installed xp in a computer after total formatting of full hard disc.
then installed open office .org . No ms office. Both ddd and dsnap
worked!
previously we found, without ms-office, these tools do not work. looks
like ooo has some internal engine just like ms-office. (i don't know
anything about programming).

I made a pic. see it.



---------- JAVA BASED DSNAP , TRADE TIME DATA CAPTURE, TILL NOW DATA DURING TRADE HOUR -----------

We capture trade time data from these links:

http://www.dsebd.org/latest_share_price_scroll_l.php (THIS ONE IS BEST)
http://www.dsebd.org/latest_share_price_scroll_by_ltp.php
http://www.dsebd.org/latest_share_price_alpha.php

here is a problem. for this we can't use it as eod data at the end of
trade hour. the problem is 'there is no open price'. we take ycp as
'open'. ycp means 'yesterday closing price'. We hope dse will give us
'open' someday. till then no way. we will work with wrong data. IS THERE ANY WAY TO CAPTURE ''TODAY'S FIRST TRADED PRICE''? WE WANT TO GET RID OF WRONG DATA. ONE OF MY FRIENDS SAID, IT IS POSSIBLE. BUT HOW? DON'T KNOW YET... waiting ...

see the attached pic. I kept 2 buttons: one for 'with index' another
for without index. index data will be captured from this link:
http://web.dsebd.org/admin-real/index-graph/companygraph.php?graph_id=gen
hi and lo is there. but where is open close volume? in the home page:
http://www.dsebd.org/index.php

Remember, volume means 'value in taka' in case of general index.
also see, in the home page, there is yesterday's index data. take ycp
as open. 'Latest' is the 'close'. volume is value in taka so far.

we will take the latest price or last traded price as 'close' in dsnap.
hi means highest so far.
lo means lowest so far.
open means ycp.
volume means volume. total volume so far.

note, we will not put 'time' in our output data file. we r not making
intraday charts. look at a csv file generated by dsnap.

Please keep 2 options, one for 'with index' and another for 'without index', as I showed on the sample image. 'Without index' will help us when internet connection is too slow.


=========== INTRADAY ===============
you can make another tool for intraday data. there will be time stamp.
five min interval or one min interval etc. date and time both will be
kept in the csv file. we do not need this tool right now. O-H-L-C-V
all will be inside that time interval, not ''till now''.

==========================

AMIBROKER
Is there any software against amibroker? for linux. www.amibroker.com
we use this software for watching graphs or charts. Now we need a linux based amibroker.


============ VOLUME IN DSEGEN ==========

by the way, don't forget to divide the index-volume by 100.

i mean, omit last 2 digits, also omit digits after the decimal.

example
when dse's total trade value in taka is 11002837568.45 tk, then our
output will be 110028375 as volume.
this rule is for 00dsegen only. for both ddd and dsnap.

2 files attached. these r our output csv files. one is from dsnap,
other is from per-day-eod.

----------------- -------------

Note:

in eod / ddd,

only source is the mst file. I told you about the multiple address earlier. This one is primary address:
http://admin.dsebd.org/admin-real/mst.txt

we will add ''public'' and ''spot'' transactions, and we will omit
''odd-lot'' and ''block'' transactions.

open= open
high= highest of the day, it may be in public or may be in spot. just
take the highest.
low= lowest of the day, it may be in public or may be in spot. just
take the lowest.
close= close
volume= add both volumes in public and spot. summation.

note, at the lower part of the mst, index open and close is given. hi
and lo will be collected from another page that i said earlier.



----------------- FORMAT OF CSV FILE ----------------

in both ddd and dsnap csv files, columns will be arranged like this:

TICKER NAME , DATE , OPEN , HIGH , LOW , CLOSE , VOLUME.

Date format will be dd-mm-yy

you can keep a row for 'headlines'. it is not necessary. But don't
forget to keep the serial like my example.
Headline means ''TICKER NAME , DATE , OPEN , HIGH , LOW , CLOSE ,
VOLUME'' these words written in the first row of each column.
Headline is not necessary. my attached files have no headlines. serial
is very important.

--------------- ----------------

GEN INDEX DATA FOR DDD/EOD:

OPEN= at the lower part of mst file.

HIGH= http://web.dsebd.org/admin-real/index-graph/companygraph.php?graph_id=gen

LOW= http://web.dsebd.org/admin-real/index-graph/companygraph.php?graph_id=gen

CLOSE= at the lower part of mst file.

VOLUME= at the upper part of mst file.

------- -------------- --------

GEN INDEX DATA FOR DSNAP/TRADE-HOUR:

OPEN= yesterday close. found in the homepage. (C)

HIGH= http://web.dsebd.org/admin-real/index-graph/companygraph.php?graph_id=gen

LOW= http://web.dsebd.org/admin-real/index-graph/companygraph.php?graph_id=gen

CLOSE= latest condition of index. found in the homepage. (A)

VOLUME= total volume till now. found in the homepage. Don't forget to divide it by 100. (B)



See the above image. See the A-B-C-D marks I placed.


--------------------

00DSEGEN: (in case of EOD/DDD only):

It is ''dse general index''. (not dsi or all share index or not dse20 index.).

ticker name will be 00dsegen.

open is 'yesterday close'. In case of index it is always
open=yesterday's close. Source is lower part of mst file.

high is today's highest. source is
low is today's lowes. source is
close is today's closing. source is at the lower part of mst file.

volume is 'total value in taka'. source is upper part of mst file. In
the output file we must put the data as 100th. That means, we will
delete the digits after decimal, and we will delete the last 2 digits.
For example, when total value in taka is ''142578548.25 tk''. our
output will be ''1425785'' . (see, last 2 digits before the decimal,
and all digits after the decimal is deleted. i mean, 48.25 is absent
in the output).

----------------------------

here is some hints for DDD/EOD TOOL (NOT TRADE HOUR TOOL):

See some cuttings from mst file:

========= final index means closing ==============

Final Index Change(Points) Change(%)
------------ -------------- -----------

ALL SHARES PRICE INDEX (DSI) 4445.80700 -86.57018 -1.9100391
DSE - 20 INDEX (DS20) 3049.46682 -71.68465 -2.2967372
DSE GENERAL INDEX (DGEN) 5399.64698 -103.73196 -1.8848776


========= this value(tk) means volume ===========

TOTAL TRANSACTIONS

A. NO. OF TRADES : 229777
B. VOLUME(Nos.) : 56388012
C. VALUE(Tk) : 16375146890.10


=========== opening and closing at the lower part of mst ========

INDEX NAME OPENING INDEX CLOSING INDEX

DSI 4532.37718 4445.80700
DS20 3121.15147 3049.46682
DGEN 5503.37894 5399.64698

=======================

take the red colored parts only (in the examples above).



-----------------------------------------------------

all instrument-codes (= ticker names) should be arranged
alphabetically. no ticker should come twice.

at the most right column, it is 'value'. we don't need 'value'. don't
put in your output file.
-----------------------

OUTPUT FILE-NAME:

Please keep an 'output folder' for each tool, EOD and trade-hour.

this folder will contain all the output csv files.

new files will replace the old file if the file-name is same.

file-name should be created automatically.

for eod tool, filename will be like this: eod_19-02-2010,
eod_20-02-2010, eod_21-02-2010 etc etc.

for trade-hour tool, file-name will be ''snap_19-02-2010''. Same name
will be created again and again, if i run the tool again in the same
date.


IF TRANSACTION TAKES PLACE BOTH IN PUBLIC AND SPOT MARKET ON A SAME DAY (FOR EOD TOOL):

we have to add both public and spot transactions always.

Like this...

open ----- as usual.

high ---- the highest one, it may be in public, or may be in spot.

low ---- the lowest one, it may be in public, or may be in spot.

close ---- as usual.

volume -- summation of both volumes. public volume+ spot volume.



NOTE:

1. We should not place 'non traded ticks' in our output csv file. It will create wrong data.

2. All stock exchange authorities of the world provides data as csv format, but our DSE does not. So, we need such tools.

3. We should not capture data from BIASL (or other similar sites), they may provide old or wrong data.

---------------------


No comments: