Comments on: Automated Talos Analysis http://atlee.ca/blog/2009/02/05/automated-talos-analysis/ programming, photography, media, and anything else that strikes my fancy Wed, 30 Jun 2010 13:38:49 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Mike Beltzner http://atlee.ca/blog/2009/02/05/automated-talos-analysis/comment-page-1/#comment-49337 Mike Beltzner Thu, 19 Feb 2009 16:24:34 +0000 http://atlee.ca/blog/?p=267#comment-49337 Just thought of another thing that would be useful in the output: a link to the range of checkins used for the build in which a regression was spotted. We basically have no good way of getting at that information now. Just thought of another thing that would be useful in the output: a link to the range of checkins used for the build in which a regression was spotted. We basically have no good way of getting at that information now.

]]> By: chris http://atlee.ca/blog/2009/02/05/automated-talos-analysis/comment-page-1/#comment-48954 chris Tue, 10 Feb 2009 19:13:38 +0000 http://atlee.ca/blog/?p=267#comment-48954 Mike - No, there's no channel where these notices are being pumped out experimentally for now. Stay tuned! Mike – No, there’s no channel where these notices are being pumped out experimentally for now. Stay tuned!

]]> By: chris http://atlee.ca/blog/2009/02/05/automated-talos-analysis/comment-page-1/#comment-48953 chris Tue, 10 Feb 2009 19:11:03 +0000 http://atlee.ca/blog/?p=267#comment-48953 AndersH - I'd prefer to keep the third point identified as the regression point on the graph for now. I find it useful to envision how the system would behave in real-time, when new data points are being added. You can't make the call if an out-of-bounds data point is a real regression until you've seen a few of them. Sure, after the fact you will want to know when the regression started, and I should be including the previous spikes in order to give an accurate changeset or buildid. I'm also experimenting with another method of finding the regression points based on t-tests. This seems to produce better results, in that it identifies the same regressions, but with fewer spikes. I'm expecting to use this method rather than my previous sliding standard deviation window method. AndersH – I’d prefer to keep the third point identified as the regression point on the graph for now. I find it useful to envision how the system would behave in real-time, when new data points are being added. You can’t make the call if an out-of-bounds data point is a real regression until you’ve seen a few of them. Sure, after the fact you will want to know when the regression started, and I should be including the previous spikes in order to give an accurate changeset or buildid.

I’m also experimenting with another method of finding the regression points based on t-tests. This seems to produce better results, in that it identifies the same regressions, but with fewer spikes. I’m expecting to use this method rather than my previous sliding standard deviation window method.

]]> By: Mike Beltzner http://atlee.ca/blog/2009/02/05/automated-talos-analysis/comment-page-1/#comment-48902 Mike Beltzner Mon, 09 Feb 2009 13:12:29 +0000 http://atlee.ca/blog/?p=267#comment-48902 Agreed - this has been needed for a long time. Is there a channel where the notices are being pumped out experimentally for now? Also, what sort of donuts do you want me to bring you as a reward? :) Agreed – this has been needed for a long time. Is there a channel where the notices are being pumped out experimentally for now? Also, what sort of donuts do you want me to bring you as a reward? :)

]]> By: AndersH http://atlee.ca/blog/2009/02/05/automated-talos-analysis/comment-page-1/#comment-48852 AndersH Sun, 08 Feb 2009 11:00:42 +0000 http://atlee.ca/blog/?p=267#comment-48852 For debugging, it might be useful to show the 2.5 standard deviation limit as a shaded area around the mean. Because you wait to the third spike to identify a regression, your regressions are marked to late. When you identify a regression, you should move it back to the first spike within the deviation interval. It seems, you include the data for spikes and regressions in the calculation of the deviation. You might not want to do that. I assume this it is the reason the data for Jan 31, 8:00 is not marked as a spike. For spikes you could do that simply by not using data points in the calculation. For regression you could find the size of the "jump" by comparing the mean before the first spike of the regression and the three spikes in the regression, and then moving all data after the regression by this amount for the purposes of deviation calculation. You should probably reinitialize the mean after a regression with the mean of the three spikes in the regression, so the data from e.g. Jan 30, 13:34 wont be identified as a new regression. For debugging, it might be useful to show the 2.5 standard deviation limit as a shaded area around the mean.

Because you wait to the third spike to identify a regression, your regressions are marked to late. When you identify a regression, you should move it back to the first spike within the deviation interval.

It seems, you include the data for spikes and regressions in the calculation of the deviation. You might not want to do that. I assume this it is the reason the data for Jan 31, 8:00 is not marked as a spike. For spikes you could do that simply by not using data points in the calculation. For regression you could find the size of the “jump” by comparing the mean before the first spike of the regression and the three spikes in the regression, and then moving all data after the regression by this amount for the purposes of deviation calculation.

You should probably reinitialize the mean after a regression with the mean of the three spikes in the regression, so the data from e.g. Jan 30, 13:34 wont be identified as a new regression.

]]> By: Justin Dolske http://atlee.ca/blog/2009/02/05/automated-talos-analysis/comment-page-1/#comment-48839 Justin Dolske Sun, 08 Feb 2009 04:58:58 +0000 http://atlee.ca/blog/?p=267#comment-48839 Oooooh, FANTASTIC. We really, really, really need something like this, so it's awesome to see some progress on this front. Oooooh, FANTASTIC. We really, really, really need something like this, so it’s awesome to see some progress on this front.

]]>