Building a Genesis Log File Grapher website for me and community - need feedback!

[jay_klay_studio]

I am a software engineer by day, and potter by night. Just got an Olympic electro sitter with the Bartlett Genesis 2.0 controller. I've been saving my log files, but they're hard to read, and without some formulas to apply to the .csv, there's a lot of info missing. Thought I'd build something for myself, and while I'm at it share it with the community.

In theory, you'll upload the .csv log file and then be presented with a line graph of the firing, with lines representing actual and target just like the Genesis graph (but ya know, bigger than a postage stamp). In addition, I'd like to provide more info in a table detailing the firing.  I know there are some missing events from the 3 log files I've pulled so far. Errors, aborted firing... ? What else? Here are the things that I've already come up with. 

General info:
start date/time
All info values
program name
diagnostics?
Total time (advance a counter for every t30 with a number in it)
firing complete info and date/time

In a table:
segments (each of these would be a column)
- type? Like "start ramp" for regular segment changes, "start hold" for holds, "skip step" for skips
- create array as we map
- start date/time
- segment number
- segment hold time
- startTemp
- total time per segment
- actual average ramp speed
- setpoint average ramp speed (is this always the same as target ramp, or does the genesis adjust setpoint on the fly?)
- target ramp speed
- any errors that occured during this segment (what does this look like?)
 

If anyone has a recent log file they could send me that might have other entries that I'm missing, that would help. Or some advice on things I should add? @neilestrick maybe?

Edited December 2, 2023 by jay_klay_studio
clarify

[jay_klay_studio]

Thought this would be a good resource for folks with the Genesis. If anyone has any thoughts on ways to get the word out once it's finished, LMK. BTW, here's a progress shot. The line graph is done. I have all of the data parsed from the rest of the CSV as well just need to build the table. 

And in case you're wondering, that is a real log file from my last firing (I decided part way into the candle to skip to the next segment).

Edited December 5, 2023 by jay_klay_studio

[PeterH]

Might also be worth also offering a (target-actual) plot. People tend to "see" the distance normal to the lines, rather than the distance parallel to the y-axis.

[jay_klay_studio]

56 minutes ago, PeterH said:

Might also be worth also offering a (target-actual) plot. People tend to "see" the distance normal to the lines, rather than the distance parallel to the y-axis.

Yeah the nice thing about this chart library is that when you hover, you get that tooltip (floating box in screenshot) and a vertical line (and two dots on either graph line) that shows you how they line up. But I agree that the most important info here is the delta of target vs actual (other than when you skip a segment of course). I was planning on including an average ramp (deg / hour) deviation for each segment in the data table. But graphing it might be more helpful!

Edited December 5, 2023 by jay_klay_studio

[neilestrick]

So are you looking for firing log files that have specific events and types of error codes in them? I'm afraid I don't have anything beyond normal firings. I haven't had an error code on my kilns in a long time.

[Dick White]

I've been doing this manually the hard way for years, using a spare thermocouple poked through a spyhole and a logging pyrometer connected to a laptop running in real time beside the kiln. I imported the .csv file into Excel and beat on it from there. It is so much easier now with the log file built into the Genesis, plus there is more information collected.

My kilns are an L&L and a Skutt modified for zone control (and with a real Genesis controller, not the Skutt dumbed down touchscreen), so the controllers are collecting data for all thermocouples. In theory, the controller is keeping all zones reasonably on track with each other, but if one goes astray, graphing each zone will show the anomalies.

My Genesis log files contain a column for the output power to each zone shown as a percent on-time during that interval. I rarely graph that, but it is useful data to know how the sections are performing. The one time I did graph it, it was to diagnose another kiln and I was able to show the owner where the kiln maxed out at 100% on-time but the temperature rose only slowly - time for an element change.

The log contains the expected settings for the segment (ramp rate, target temp) at the beginning of that segment, which would be where you got your green line. However, as the segment progresses, The kiln may or may not actually maintain that expectation, particularly in the final segment(s). The controller is recalculating interim setpoints every few minutes as it goes, which are listed in the log for each time interval. When the elements begin to lag, your blue line will begin to deviate from the green line. At that point, either the continuous green line becomes useless, or you need to insert a discontinuity  where the as-programmed line for segment X ends and the as-programmed section for segment Y begins so that both the as-programmed segment and actual segment align on the timeline. In your sample, the kiln seems to be doing nicely as shown by the slopes of the lines for the various segments are parallel (and would be more or less concurrent had you not chopped the initial hold). Worn elements show a very different pattern - don't ask how I know that (or maybe do ask and I'll find a chart I did for a troubled kiln some years ago).

You mentioned calculating ramp rates (degrees per hour). Yes, this is useful data, but I've found it tricky to calculate in a uniformly meaningful way. The Genesis logs every 30 seconds. Calculating an instantaneous ramp rate at each data point is all over the place. The kiln takes time to accept the heat input, and continues heating for a bit after the controller clicks off. Thus, you need to be calculating a rolling average over some small number of previous minutes. This data is particularly useful to know what the actual kiln performance was during the final segment coming into cone vs. the Orton/controller's specified rate (e.g., 120℉/hour at medium speed). The Bartlett controllers have long contained an adaptive algorithm licensed from Orton in the cone-fire mode that shifts the final target up or down based on the actual ramp rate being achieved during the final segment, so one's firings tend to cause the appropriate cone bend even as the elements wear and can't maintain the ramp. So many users don't notice it until the firings become too long (if they are keeping a firing log) or the controller throws the dreaded E-1. As we have discussed elsewhere in this group, that's a problem when setting off custom ramp-hold programs. If your kiln can't actually maintain the specified ramp, the controller will keep pushing to the specified final temperature, and now the expected cone is overfired.

Carry on the good work

dw

[neilestrick]

46 minutes ago, Dick White said:

So many users don't notice it until the firings become too long (if they are keeping a firing log) or the controller throws the dreaded E-1.

Correct. The firing may be using more energy to maintain the firing speed, but you won't see it in the total firing time until it gets bad enough to show up there. The easiest way to tell if firings are using more energy is to check the firing cost. Even if the dollar amount isn't accurate to real world costs, you just need a baseline to compare to.

[PeterH]

Am I having a senior moment, or does this graph have teething troubles?

Shouldn't the major discontinuities in the  target temperature line be at the segment-start times?
... and should the two peak temperatures differ by ~60mins?
... and did skipping part of the candle confuse the analysis?

[neilestrick]

3 minutes ago, PeterH said:

Shouldn't the major discontinuities in the  target temperature line be at the segment-start times?
... and should the two peak temperatures differ by ~60mins?
... and did skipping part of the candle confuse the analysis?

I think maybe you're assuming that the kiln is staying exactly on track with the program? Unless it's a fairly slow program or a small but powerful kiln, it rarely does.

[Dick White]

1 hour ago, PeterH said:

Am I having a senior moment, or does this graph have teething troubles?
 

In senior times, teeth are falling out, not growing in. Don't ask how I know this. The difference between the two lines is attributable to his skip segment during the initial preheat hold. If you mentally extend that flat section of the blue line so that both lines concur at the first serious ramp up (i.e., move the blue inflection at minute ~80 to match the green inflection at minute 180), the two lines will be close up until ~1800 degrees. At that point, the slope of the blue line in segment 5 should continue straight until a distinct inflection at the beginning of segment 6 (as shown on the green line) but instead the ramp rate begins to falter, likely due to somewhat worn elements. There is not a distinct inflection in the blue line at the ramp change for the beginning of segment 6, instead it gently curves from one to the next. Even with the programmed slower ramp rate in segment 6, the kiln isn't keeping up. The slope of the blue line, while steady, is slightly lower than the slope of the green line and the blue segment 6 takes a bit longer to complete than the green. These are the kind of diagnostics that I find such graphs to be helpful.

[jay_klay_studio]

17 hours ago, Dick White said:

My kilns are an L&L and a Skutt modified for zone control (and with a real Genesis controller, not the Skutt dumbed down touchscreen), so the controllers are collecting data for all thermocouples. In theory, the controller is keeping all zones reasonably on track with each other, but if one goes astray, graphing each zone will show the anomalies.

@Dick WhiteDo you have a log file with multiple TCs you could send me? I only have one TC. Also, in your opinion, would it be helpful to be able to graph each TC separately as a line on the chart? Or, would it make more sense to be able to choose which TCs are used and the average temp from those TCs used for the single line?

17 hours ago, Dick White said:

My Genesis log files contain a column for the output power to each zone shown as a percent on-time during that interval. I rarely graph that, but it is useful data to know how the sections are performing. The one time I did graph it, it was to diagnose another kiln and I was able to show the owner where the kiln maxed out at 100% on-time but the temperature rose only slowly - time for an element change.

Is this the "out#" column? I had wondered about that. It does appear that when I start to drop away from the target ramp that my out#'s are at 100, which makes sense. Not sure the relationship between TCs and Zones, but I'm guessing there's not a 1:1 there.

17 hours ago, Dick White said:

The log contains the expected settings for the segment (ramp rate, target temp) at the beginning of that segment, which would be where you got your green line. However, as the segment progresses, The kiln may or may not actually maintain that expectation, particularly in the final segment(s). The controller is recalculating interim setpoints every few minutes as it goes, which are listed in the log for each time interval. When the elements begin to lag, your blue line will begin to deviate from the green line. At that point, either the continuous green line becomes useless, or you need to insert a discontinuity  where the as-programmed line for segment X ends and the as-programmed section for segment Y begins so that both the as-programmed segment and actual segment align on the timeline. In your sample, the kiln seems to be doing nicely as shown by the slopes of the lines for the various segments are parallel (and would be more or less concurrent had you not chopped the initial hold). Worn elements show a very different pattern - don't ask how I know that (or maybe do ask and I'll find a chart I did for a troubled kiln some years ago).

I think that my elements are actually worn, and my last 200 degree ramp is off by about 20 deg/hr, which has been causing me to overfire. Kind of the original impetus to make this tool. But I hear you on the reduced utility of the two lines when they start to deviate, especially as the end of the firing is the most crucial and is where you'd have the largest X deviation. I'm thinking based on your comment to provide a little dropdown menu that can align start segments (default to start of firing), so that say you select segment 6, and the graph moves the two lines so that they start at the same time. This way, it'd be easy to tell differences in ramp, and could put you back on track in case you skipped a segment like I did.

17 hours ago, Dick White said:

You mentioned calculating ramp rates (degrees per hour). Yes, this is useful data, but I've found it tricky to calculate in a uniformly meaningful way. The Genesis logs every 30 seconds. Calculating an instantaneous ramp rate at each data point is all over the place. The kiln takes time to accept the heat input, and continues heating for a bit after the controller clicks off. Thus, you need to be calculating a rolling average over some small number of previous minutes. This data is particularly useful to know what the actual kiln performance was during the final segment coming into cone vs. the Orton/controller's specified rate (e.g., 120℉/hour at medium speed). The Bartlett controllers have long contained an adaptive algorithm licensed from Orton in the cone-fire mode that shifts the final target up or down based on the actual ramp rate being achieved during the final segment, so one's firings tend to cause the appropriate cone bend even as the elements wear and can't maintain the ramp. So many users don't notice it until the firings become too long (if they are keeping a firing log) or the controller throws the dreaded E-1. As we have discussed elsewhere in this group, that's a problem when setting off custom ramp-hold programs. If your kiln can't actually maintain the specified ramp, the controller will keep pushing to the specified final temperature, and now the expected cone is overfired.

Carry on the good work

dw

So far, the "actual" ramp rate I have calculated is based off of an entire segment. It's essentially just the temp diff between the start of the segment and end, divided by the no. of hours (based on adding up the half minute ticks for a segment in the log). Would it be more helpful to have the "actual" calculated ramp in smaller increments than just the entire segment?

Edited December 6, 2023 by jay_klay_studio

[PeterH]

59 minutes ago, neilestrick said:

I think maybe you're assuming that the kiln is staying exactly on track with the program? Unless it's a fairly slow program or a small but powerful kiln, it rarely does.

 

I freely defer to your knowledge in this area, but wouldn't 60 mins be a bit excessive?

Note that the actual temperature is shown as significantly higher than the target temperature for the first ~600 mins. With the actual temperature shown peaking ~60 mins before the target peak temperature.

So maybe the actual/target labels are the wrong way round.

Switching the labels means that the registration of several kinks in the purple line (now the target temperature) with segment changes seems very plausible.

But the two curves still seem (to me) to have registration problems, which might relate to skipping part of the program. I'm unsure if this could result in  mis-attribution of the timings of readings on one of the curves. Of course then then thermal lag has to be allowed for.

.

[jay_klay_studio]

16 hours ago, neilestrick said:

So are you looking for firing log files that have specific events and types of error codes in them? I'm afraid I don't have anything beyond normal firings. I haven't had an error code on my kilns in a long time.

A normal firing might still be helpful if you have one handy! I only have one TC and one zone, no fan control, and the only custom program I've input was fairly simple. 

[jay_klay_studio]

2 hours ago, PeterH said:

Shouldn't the major discontinuities in the  target temperature line be at the segment-start times?
... and should the two peak temperatures differ by ~60mins?
... and did skipping part of the candle confuse the analysis?

Yeah, the skip here really skewed the graph on this one. I have a plan for fixing that (see my response to Dick above).

The confusing thing about the previous image I posted is that segments 2-7 took longer (especially segment 6 to peak), because my elements are worn as Dick mentioned. But because I skipped about 75% of the candle, it shifted the "actual" graph line back by about 1.5 hours. You'll notice that it's slower rate, though, means that it's only about an hour off from the target at peak temp (so, not taking into account the skip, the actual firing took about 30 minutes longer).

Here's a graph from my most recent firing (where it's clear that my kiln struggled to maintain ramp at peak but otherwise was pretty close): 

Edited December 6, 2023 by jay_klay_studio

[Piedmont Pottery]

Will your program accept multiple log files for long firing time programs?  I often have two, sometimes three log files per firing, depending on the program.  Also, I'm hoping that your program will be compatible with first generation Genesis log files.  I'm really looking forward to your finished product.

[jay_klay_studio]

5 minutes ago, Piedmont Pottery said:

Will your program accept multiple log files for long firing time programs?  I often have two, sometimes three log files per firing, depending on the program.  Also, I'm hoping that your program will be compatible with first generation Genesis log files.  I'm really looking forward to your finished product.

@Piedmont Pottery I didn't know that multiple files was a thing! I'm new to the Genesis world (just got my first and it's a 2.0). And I'd love to be able to support other generations if possible. It'd actually be really helpful to know if Bartlett has adjusted their log file format much since the first generation. That might give me some clue as to how long this tool will be useful before I need to do some maintenance to keep up with log file format changes...

I don't suppose you could attach one of your multi-log files here?

[Dick White]

25 minutes ago, Piedmont Pottery said:

I often have two, sometimes three log files per firing, depending on the program.

oops, I take all that back. I just looked for the log file when I first noticed the timer can only show 24 hours, and indeed as you said, there are 2 files - one showing the first 24 hours of the preheat hold and another picking up right from there through the end of the firing.

Edited December 6, 2023 by Dick White
added a sentence

[Dick White]

21 minutes ago, jay_klay_studio said:

I don't suppose you could attach one of your multi-log files here?

They are discrete .csv files that must be appended together in your program.

[jay_klay_studio]

5 minutes ago, Dick White said:

They are discrete .csv files that must be appended together in your program.

Sorry that's what I meant. Should have said post the "set" of multi-log files.

[Dick White]

@jay_klay_studio How is the best way to send you some files? I don't think the forum will allow public posts of massive data files (or if it does, it shouldn't as a courtesy to the rest of the users...).

 

[Dick White]

1 hour ago, jay_klay_studio said:

So far, the "actual" ramp rate I have calculated is based off of an entire segment. It's essentially just the temp diff between the start of the segment and end, divided by the no. of hours (based on adding up the half minute ticks for a segment in the log). Would it be more helpful to have the "actual" calculated ramp in smaller increments than just the entire segment?

Yes, I use either a prior 5-minute or 10-minute rolling average. If you did that, you would see exactly where the elements begin to falter in mid-segment as the temperature increases, and how bad it gets as the firing progresses.

[jay_klay_studio]

13 minutes ago, Dick White said:

@jay_klay_studio How is the best way to send you some files? I don't think the forum will allow public posts of massive data files (or if it does, it shouldn't as a courtesy to the rest of the users...).

 

@Dick White I would defer to people who've been on the forum for awhile. I will say, however, that despite the number of entries in a log file, CSVs are small by nature. This file that I'm looking at has somewhere around 10,000 points of data (cells), but the file is only 33kb (which is smaller than any image one might be inclined to post). If you want to email me to be safe, my address is joshua@young.net

[PeterH]

How do you determine the start-time of the segments? Is it just when the end-temperature is reached in the CVS file?

PS Minor point, but you might try changing the line colours to additive ones, which might show overlapping lines more clearly. Perhaps cyan & magenta, as some real-world yellows are a bit dodgy. The idea has some history in statistical presentations.

 

Edited December 7, 2023 by PeterH

[Bill Kielb]

20 hours ago, jay_klay_studio said:

Here's a graph from my most recent firing (where it's clear that my kiln struggled to maintain ramp at peak but otherwise was pretty close

Much better! I would suggest to see percentage output from the controller if possible,  by zone would be great, so at a glance I can tell if the kiln is under powered at some point or the controller PID needs some calibration and tweaking s practical.

[High Bridge Pottery]

There's 1000kb limit per post but most csv files should be way under that.

 

I would prefer the x-axis in hours and another option for easier seeing of the firing is to dash the setpoint line so it covers less of the firing curve.