Commit Graph

28 Commits (838db689ad7787868ac55645352e1523544aebfc)

Author SHA1 Message Date
Mike Gerwitz 2a84e44a58 bin/tame: Fix runner output line clearing
The output was being omitted under certain conditions, meaning that users
would have to look in the runlogs for errors.
2022-01-28 09:21:34 -05:00
Mike Gerwitz 8b255c2251 tame: tamed --help: Add missing closing quote to awk example 2022-01-26 13:51:34 -05:00
Mike Gerwitz 8fbddfb3b3 tamed: Fix --help and add another reporting example
$2 was not escaped and would fail expansion.  I apparently did not run
--help before committing.  Shame on me.
2022-01-20 23:32:28 -05:00
Mike Gerwitz 6fd570477a tamed: Add runtab and TAMED_RUNTAB_OUT
This provides logging that can be used to analyze jobs.  See `tamed --help`
for some examples.  More to come.

You'll notice that one of the examples reprents package build time in
_minutes_.  This is why TAMER is necessary; as of the time of writing, the
longest-building package is nearly five and a half minutes, and there are a
number of packages that take a minute or more.  But, there are potentially
other optimizations that can be done.  And this is _after_ many rounds of
optimizations over the years.  (TAME was not originally built for what it is
currently being used for.)
2022-01-19 16:47:12 -05:00
Mike Gerwitz 4a3b86f480 tamed: Ignore SIGUSR2
This was originally going to tell tamed to redraw the runner status line,
but a different approach was taken.
2022-01-19 15:41:28 -05:00
Mike Gerwitz c72d908a3f tamed: Add missing --report to help
Missing from previous commit.
2022-01-19 13:29:23 -05:00
Mike Gerwitz 756dcd7894 tamed --report and runner status line (TAMED_TUI)
This is something that I've wanted to do for quite some time, but for good
reason, have been avoiding.

`tamed --report` is fairly basic right now, but allows you to see what each
of the runners are doing.  This will be expanded further to gather data for
further analysis.

The thing that I was avoiding was a status line during the build to
summarize what the runners are doing, since it's nearly impossible to do so
from the build output with multiple runners.  This will not only allow me to
debug more easily, but will keep the output plainly visible to developers at
all times in the hope that it can help them improve the build times
themselves in certain cases.

It is currently gated behind TAMED_TUI, since, while it works well overall,
it is imperfect, and will cause artifacts from build output partly
overwriting the status line, and may even occasionally clobber the PS1 by
erasing the line.  This will be improved upon in the future; something is
better than nothing.
2022-01-19 11:51:48 -05:00
Mike Gerwitz 2e50af1220 Copyright year update 2021 2021-07-22 15:00:15 -04:00
Mike Gerwitz 6221ce5fee bin/tame (verify-runner): Add missing id param
This was referencing a global $id, which is not the value we are interested
in (and may not exist at all).  Add the missing param.
2021-03-12 14:14:42 -05:00
Mike Gerwitz 7325578624 bin/tame{,d}: Fix assignments that lose exit code
Ensure that we fail if the command in the assignment fails.
2021-03-12 14:14:40 -05:00
Mike Gerwitz e7e4a61cf4 bin/tame: read {=>-r} where missing
While these should not be necessary in practice, there's no reason _not_ to
do this.
2021-03-12 09:47:38 -05:00
Mike Gerwitz 2a56c75345 bin/tame: Remove unnecessary trailing backslashes
This was originally in a Makefile, long ago, where backslashes were
actually needed.
2021-03-12 09:47:34 -05:00
Mike Gerwitz da7a2c71c7 tamed: TAMED_JAVA_OPTS: New environment variable
This will be passed to dslc and then to the JVM.  The intent is to permit
fine-grained heap ratio tuning.
2020-08-19 10:19:04 -04:00
Mike Gerwitz bfea768f89 Copyright year 2020 update 2020-03-06 11:05:18 -05:00
Mike Gerwitz 0718d80257 bin/tamed: Fail without explicit DONE
We want to fail e.g. on a JVM crash.
2020-01-21 11:40:14 -05:00
Mike Gerwitz 6f5796238a tame: Create guard parent directory
It may not exist on certain systems (e.g. build containers).
2019-04-04 14:52:36 -04:00
Mike Gerwitz 1a35232bd8 Parallel build support
tamed was originally designed with support for parallel builds in mind, but
I hadn't completed that work because we didn't have enough hardware that
we'd benefit strongly from it.  That has since changed.

tamed will now spawn additional runners as needed to fulfill requests, which
works around the issue of not knowing how many jobs GNU Make is going to try
to do at once.

There were a couple minor dependency fixes/workarounds for now in the
Makefile, but otherwise everything appears to be working great.
2019-04-04 14:41:07 -04:00
Mike Gerwitz e022a3133d Copyright year simplification and update to Ryan Specialty Group
This now uses year ranges, which I'll update annually.

This also renames "R-T Specialty" to "Ryan Specialty Group".  The latter is
the parent company of the former.  I was originally employed under the
former when LoVullo Associates was purchased, by I now work for the parent
company.
2019-02-07 13:23:09 -05:00
Mike Gerwitz f62f2ccacb bin/tame: Shift arguments after -v
* bin/tame (main): Fix issue where -v did not shift arguments.
2018-12-04 13:45:36 -05:00
Mike Gerwitz cd5440b8da tamed: Clarify usage output shell example
* bin/tamed (usage): Clarify killing when run on a shell.
2018-12-03 16:46:06 -05:00
Mike Gerwitz 079d1dcfaf tamed: Do not stall if TAMED_SPAWER_PID is running
This will ensure that tamed does not stall while e.g. make is still
running.  This makes TAMED_STALL_SECONDS almost useless; maybe it'll be
removed in future versions.

* bin/tame (TAMED_SPAWNER_PID): Export variable.
* bin/tamed (TAMED_SPAWNER_PID): New variable, default to PPID.
  (spawner-dead): New function.
  (stall-monitor): Use it.
  (usage): Update documentation.
* build-aux/Makefile.am: Set TAMED_SPAWNER_PID to own id and export.
2018-12-03 16:25:25 -05:00
Mike Gerwitz 5cb78cc47d dslc: Invoke with static rater path
This frees us from requiring a rater/ directory in the working
directory.  However, it is important that we continue using it if it
exists, since there are additional things that haven't yet been moved
into the tame repo.

* bin/dslc.in: Provide path to rater/ directory.
* src/current/src/com/lovullo/dslc/DslCompiler.java: Use provided rater/ path.
2018-11-08 09:26:07 -05:00
Mike Gerwitz 8fa7d9ece6 bin/tame: Inherit TAME_CMD_WAITTIME from environment
* bin/tame (TAME_CMD_WAITTIME): Renamed from `RUNNER_CMD_WAITTIME'.
    Inherit from environment, default 3.
  (command-runner): Sleep for an additional TAME_CMD_WAITTIME seconds after
    requesting runner reload to give more time in case of high load.
  (verify-runner-ack): Rename variable.
  (usage): Document env var.
* build-aux/Makefile.am: Export TAME_CMD_WAITTIME.
2018-10-19 10:15:14 -04:00
Mike Gerwitz a4c8c0d840 bin/tame: Better runner re-try
Try to re-post message, since the previous message will have already been
read (otherwise the previous echo would have hung).

* bin/tame (EX_STALLED): New exit code.
  (command-runner): Re-post message after stall.  If unrecoverable, provide
    a more clear error and exit with EX_STALLED.
2018-10-16 22:23:57 -04:00
Mike Gerwitz b7167467b0 Propagate TAMED_STALL_SECONDS
bin/tame (TAMED_STALL_SECONDS): Export variable.
build-aux/Makefile.am (TAMED_INSTALL_SECONDS): Export variable.
2018-10-16 09:26:37 -04:00
Mike Gerwitz db1c03dfd9 tame{,d}: Reload runner when unresponsive
This tries to be a bit more resilient in case a runner becomes unresponsive,
rather than waiting for tamed to kill itself.

* bin/tame (RUNNER_CMD_WAITTIME): New variable.
  (command-runner): Tell runner to reload if it does not respond in
    RUNNER_CMD_WAITTIME seconds.
  (verify-runner-ack): New function.
* bin/tamed (mkfifos): Only keep stdin open.  stdout isn't necessary, and
    may have actually been causing subtle issues.
  (spawn-runner): Support restarting dslc on SIGHUP.
2018-10-16 08:53:04 -04:00
Mike Gerwitz 6027769633 Integrate new compilation scripts, remove cqueue and Makefile.2
This is a major step toward normalcy---removing the kluge of a build process
that was causing so many issues.  Rather than echoing all operations to a
queue file before passing it off to dslc, the new build scripts in `bin/'
are used to invoke tame normally, as needed.  This solves all of the current
issues with things not rebuilding when they should.  And, as a bonus, tab
completion on targets works.

Sorry this took so long.  There wasn't much motivation until we hired so
many people that are suffering from this.

This does a few major things, along with some miscellaneous others:
  - Invoke bin/tame directly;
  - Merge Makefile.2.in into Makefile.am; and
  - Fix up some targets.

* build-aux/Makefile.2.in: Delete file.  Mostly merged with Makefile.am.
* build-aux/Makefile.am: Add a bunch of new targets and definitions from
    Makefile.2.in.  Modify all that previously used .cqueue to now invoke
    `$(TAME)' directly.  Remove miscellaneous targets for trying to proxy
    targets to Makefile.2.
  (saneout, _go): Remove definitions.
  (.NOTPARALLEL): Add to prevent parallel builds.
  (ui/program.expanded.xml)[.version.xml]: Remove dependency for now.
  (clean): Also clean generated PHP files.  Follow symlinks to clean core.
    This is still incomplete (does not clean all rate table stuff).
  (suppliers.mk)[xmlo_cmd]: Remove.  See `gen-make' and `gen-c1make'.
  (lvroot)[summary-html]: New dependency.
  (kill-tamed, tamed-die): New targets (former alias of latter) to kill
    tamed.
* build-aux/gen-c1make: Generate `$(TAME)' invocation.
* build-aux/gen-make: Likewise.  Remove `xmlo_cmd' output.  Ignore recursive
    `tame' symlink (this can be removed once we clean `rater/' up.
* build-aux/m4/calcdsl.m4 (TAME): Update description to reflect that it
    should now be the path to `bin/tame'.  Adjust `AC_CHECK_FILE' lines
    accordingly.
  (tame_needed_ver): Remove.  We have been in the same repo as TAME itself
    for quite some time.  Remove associated code.
  (AC_CONFIG_FILES): Remove `Makefile.2'.
* src/current/src/com/lovullo/dslc/DslCompiler.java (_DslCompiler)[compile]:
    Perform validation prefore `compile' command rather than a separate
    `validate' step.  Remove `rm'.
  [compileSrc]: Stop echoing command.  This was only necessary because of
    the previous Makefile klugery; now Make echoes on its own correctly.
2018-10-11 22:25:18 -04:00
Mike Gerwitz cf57857ce5 bin/: Server/client build scripts
These scripts allow the TAME compiler stack to be invoked naturally, rather
than requiring the use of a Makefile today.  This will not only allow users
to more easily invoke the compiler, but will also allow us to invoke TAME
naturally from Makefile and remove the klugery that has existed for so
long.

This users a server/client architecture in order to mitigate the startup
cost of the JVM.  More documentation will follow.

Note that there are a bunch of symlinks in rater/---this is a transition
step to allow the build to continue working as it did before, which relies
on a directory structure that exists outside of this repository.  This will
be cleaned up in the future.

* .gitignore (bin/dslc): Add ignore for generated file.
* bin/dslc.in: New script to encapsulate Java invocation.
* bin/tame: New script (client).
* bin/tamed: New script (server).
* configure.ac (JAVA_OPTS, DSLC_CLASSPATH, AUTOGENERATED): New variables for
  dslc.in.  Output bin/dslc.
* rater/README.md: Note that this symlink mess is temporary.
* rater/c1map: New symlink for dslc assumptions.
* rater/c1map.xsl: Likewise.
* rater/calc.xsd: Likewise.
* rater/compile.xsl: Likewise.
* rater/compiler: Likewise.
* rater/dot.xsl: Likewise.
* rater/include: Likewise.
* rater/link.xsl: Likewise.
* rater/standalone.xsl: Likewise.
* rater/summary.xsl: Likewise.
* rater/tame: Likewise (warning: circular symlink).
* src/current/src/com/lovullo/dslc/DslCompiler.java (_DslCompiler)[compile]:
  Output `DONE' lines.
2018-10-08 23:25:02 -04:00