ALDL Recorded Runs File accessability

Submit feature suggestions for future versions of TunerPro here.

Moderators: Mangus, robertisaar, dex

Post Reply
SyTyArchives
Posts: 8
Joined: Sat Jul 17, 2004 4:17 pm

ALDL Recorded Runs File accessability

Post by SyTyArchives »

I'm new to tuner pro, comeing from a world of datamaster and diacom, I had one suggestion on a topic that TTS has been unresponsive on.

To help community tuning and enable people to get asistance over the net, I've been thinking up some ideas on building a web application that could read and graph ALDL data recorded from a program like this. I was hoping to have some success with Datamaster, but thier native file format is useless, and their export tool could use some work.

What I'd really lilke to see is the native file format be text based, possibly xml, something that would be easily viewable/parseable by web enabled programing platforms.

Given an easily parsable format, users could upload and share files without the need to have the diagnostic software installed. We could do some interesting stuff with data from different runs or different vehicles, and beef up the community and of course, the popularity of a particular tool.

If you have any input or any ideas, I'd be glad to work with you, as it looks like you have a lot on your plate for such a powerfull software package.

Of course, I'm mostly concerned with supporting my Sy/Ty buddies, but once the software is written, I wouldnt have a problem sharing it for other community host's to use.

Great stuff, and I hope to soon be tuning with Tuner Pro and a shweet autoprom.
User avatar
Mangus
TunerPro Author
Posts: 1918
Joined: Wed Mar 19, 2003 1:49 pm

Post by Mangus »

Thanks for the kind words. TunerPro, in its current released form, will export a log session to CSV, which is text based and very easily parsed.

A future version will allow you to customize what gets exported.

Most logging applicaitons support CSV export. I'm wondering why this is inadequate. Logs are represented in a similar way in memory and in the proprietary file formats (only in binary form), so technically everything is there.
***************************************
TunerPro Author
1989 Trans Am
SyTyArchives
Posts: 8
Joined: Sat Jul 17, 2004 4:17 pm

Post by SyTyArchives »

I'm aware of the CSV output of Datamaster, but useability is really where its lacking. ( I cant refer to tuner pro much as I havnt been able to use it yet)

By default, I'd like to see the dataloggers save out all recorded data ( and some environment data) in a standard structure. Datamaster lacks where it allows you to choose each field, but it takes 15 minutes to select *most* (cant save out every field ) of the fields for output. Given there is no standard to this format, it would be difficult to compare files, not to mention novice users may need instructions just to output usefull data.

(It looks like Tuner Pro just saves all the data ? )

Not to mention that datamaster cant then read that file back in and use it itself.

Ideally, one could export the aldl stream to a given text format (CSV is fine). Users could then upload to a site with this software and it could translate as needed. They could also download the runs to refer to later on, by opening them in their datalogging program.

I think its rather obvious a system like this would help the community a lot. I've been turning it over and Im getting some ideas and hope to be able to move on them.

I can read dm csv files, but getting compareable output will rely on the users input, which is exactly where we cant really rely on anything consistant.

I'm really just trying to make this as easy as possible for novices and the experienced who dont want to have to jump through hoops to help out.
User avatar
Mangus
TunerPro Author
Posts: 1918
Joined: Wed Mar 19, 2003 1:49 pm

Post by Mangus »

You might consider some reasons why the various logging software doesn't playback from CSV files.

CSV files contain nothing but raw text (and commas). There is standardization to it. Once a log is exported, a user can do whatever they please with the arrangement, sorting, etc, of the CSV in excel.

How could the logging program possibly discern (programmatically) that column 3 is the sample number, or column 14 is the timestamp of the sample? Its very difficult without standardization. And the point of CSV is to get rid of the standardization (read "proprietary") of the native logging format.

Its a very difficult problem to solve. Its much like trying to write a program that takes for input Moby Dick in random order (words and chapters rearranged) and outputs the novel as it was written. Pretty tough, huh?

That said, have you seen Panalyze? Its here:

http://peter.kod.nu/panalyze

And finally, I'm more than willing to publish my file format information, but I can tell you that its not simple (nor is it super complex). It won't be simple-parser friendly. Especially on the web. ;-)
***************************************
TunerPro Author
1989 Trans Am
SyTyArchives
Posts: 8
Joined: Sat Jul 17, 2004 4:17 pm

Post by SyTyArchives »

Thanks for the input, I'll have to do some more thinking on the subject once I get a chance to spend some more time planning the project.

I felt that the CSV's did lack some usefullness, and can easily see what your reffering to, but something with a well thought out structure could work.

The only alternative I can think of is if we started building a sort of 'standard data logging file format' and outline it much like the w3c does with it's document standards. Although old software wont support it, we could get new software into a realm that would. This would open up some of the posibilites with supporting software.

I know youve got a lot to do with tuner rt, but if your interested I could start laying some groundwork on the subject. Of course, I'd get feedback from some of the other datalogging software so hopefully we can cover everyone's needs.

I dont know if you have much experience with xml, but the more I work with it (and see it being used) I find it as a good medium, especially for the web. I'm open to other ideas, but thats what I keep going back to, especially with all the xml parsing stuff thats out there now.
turbodig
Posts: 10
Joined: Mon Jan 05, 2004 3:21 pm

Post by turbodig »

For anybody who might be interested in such things...

The DataMaster .uni format is not impossible to decipher. If you look at a .uni in a Hex editor, you can pick out regular frame data intervals.
There's a separator series of 00 01 02 03 04 05 06 07 08 09 0a...
right in front of the frame header data.

Diacom is even easier to figure out, and has been decoded. Ron G. did the
"datadump" C program years ago, and it was pretty straightforward.

Ian, you could probably write a CGI or perl thing to read in a native DM
or Diacom (Tunerpro too!) file and display it graphically in a web page.
This would be super-cool.

Later,

Dig
SyTyArchives
Posts: 8
Joined: Sat Jul 17, 2004 4:17 pm

Post by SyTyArchives »

Hmm, I've been resisting looking at things in hex editors, but I guess I'll have to break down eventually (especially since I plan on getting into tuning soon).

Crap, maybe It'll just be something I'll take care of on the server. One good advantage is the native files are smaller then plain text files anyway.

I guess asking everyone else to change would be harder then just handling on my end.

Dig, if you have any links that might be usefull please do send em my way.
User avatar
Mangus
TunerPro Author
Posts: 1918
Joined: Wed Mar 19, 2003 1:49 pm

Post by Mangus »

The idea of a "standard" (but powerful) logging format is a good idea. I'm down to support it, just let me know.

I can also send you the information you need for my logging format if you want to take Dig's suggestion and just make your tool import all the different formats. That will allow you to import my format.

If you engineer it, I'll comply. I'm easy going. ;-)
***************************************
TunerPro Author
1989 Trans Am
SyTyArchives
Posts: 8
Joined: Sat Jul 17, 2004 4:17 pm

Post by SyTyArchives »

Good to hear. This sounds promising and I hope to get something usefull going soon.

Keep up the good work.
Post Reply