Commit Graph

75 Commits (8948452b7112924fbbf66653eefb6b789ff3c48b)

Author SHA1 Message Date
Mike Gerwitz 1ad2fb1dc8 Copyright year update 2022
RSG (Ryan Specialty Group) recently announced a rename to Ryan Specialty (no
"Group"), but I'm not sure if the legal name has been changed yet or not, so
I'll wait on that.
2022-05-03 14:14:29 -04:00
Mike Gerwitz b90bf9d8a8 tame: build-aux/{csv2xml,tdat2xml}: Remove xml-stylesheet XML PI
These declarations are relics from when all XML files could be loaded in the
browser to render the Summary Page.  Such a thing has not worked for many
years.

The previous commit will cause files produced by these scripts to be
regenerated.

I noticed this when reading source files using XIR.

DEV-10413
2022-04-07 09:32:00 -04:00
Mike Gerwitz 8e9b2a7211 tame: build-aux/Makefile.am: Generated sources depend on scripts that generate them
This ensures that, when changes are made to these scripts, the files that
are generated from them are re-generated.

Historically this probably was not noticed because (a) they seldom changed
and (b) we had a small team and I told people to re-run bootstrapping
scripts or clean files.  The team is much larger now, and regardless,
there's no reason not to have had this in place.

DEV-10413
2022-04-07 09:31:55 -04:00
Mike Gerwitz 9fa79ce5ea TAME_PARAMS: New Makefile var
This is intended to be set via the configure script, and is being added
primarily for the upcoming flag to enable the legacy classification
system.  This is only used for the XSLT-based compiler.
2022-02-28 12:35:17 -05:00
Mike Gerwitz cd13b80f31 build-aux/check-coupling: Prohibit supplier imports of UI packages
The reverse was checked, but apparently a check for suppliers importing the
UI was never added.
2022-01-28 10:50:27 -05:00
Mike Gerwitz 6fd570477a tamed: Add runtab and TAMED_RUNTAB_OUT
This provides logging that can be used to analyze jobs.  See `tamed --help`
for some examples.  More to come.

You'll notice that one of the examples reprents package build time in
_minutes_.  This is why TAMER is necessary; as of the time of writing, the
longest-building package is nearly five and a half minutes, and there are a
number of packages that take a minute or more.  But, there are potentially
other optimizations that can be done.  And this is _after_ many rounds of
optimizations over the years.  (TAME was not originally built for what it is
currently being used for.)
2022-01-19 16:47:12 -05:00
Mike Gerwitz 756dcd7894 tamed --report and runner status line (TAMED_TUI)
This is something that I've wanted to do for quite some time, but for good
reason, have been avoiding.

`tamed --report` is fairly basic right now, but allows you to see what each
of the runners are doing.  This will be expanded further to gather data for
further analysis.

The thing that I was avoiding was a status line during the build to
summarize what the runners are doing, since it's nearly impossible to do so
from the build output with multiple runners.  This will not only allow me to
debug more easily, but will keep the output plainly visible to developers at
all times in the hope that it can help them improve the build times
themselves in certain cases.

It is currently gated behind TAMED_TUI, since, while it works well overall,
it is imperfect, and will cause artifacts from build output partly
overwriting the status line, and may even occasionally clobber the PS1 by
erasing the line.  This will be improved upon in the future; something is
better than nothing.
2022-01-19 11:51:48 -05:00
Mike Gerwitz 2e50af1220 Copyright year update 2021 2021-07-22 15:00:15 -04:00
Mike Gerwitz 416676f1ab build-aux/progtest-runner: Deterministically concatenate files by name 2021-06-09 16:10:52 -04:00
Mike Gerwitz 43204d1dd5 build-aux/Makefile.am: Lookup table dependency fix
%.xml{=>o}: %csvo rater/core/vector/table.xmlo

That is: we'll only build an object file when we try to build another object
file.  This was causing problems with dependency generation, because it will
triggering compilation early.
2021-03-17 17:02:58 -04:00
Mike Gerwitz 566d9f6536 build-aux/Makefile.am (suppliers.mk): Regenerate when any sources change
This should have been done many years ago.  This will determine if any of
the dependencies have changed for the included suppliers.mk and regenerate
it as needed, without the developer having to do so manually when imports
change.
2021-02-23 10:48:21 -05:00
Mike Gerwitz cda3e845b8 Remove verbose messages from suppliers.mk generation
* build-aux/Makefile.am (suppliers.mk): Invoke ant with `-q` to eliminate
"processing" messages for each and every file.  This also speeds up
operation slightly.
* build-aux/gen-make: Remove information echos for each file.

These changes will allow for suppliers.mk to be regenerated automatically
without being so invasive.
2021-02-23 10:47:40 -05:00
Mike Gerwitz 6f67a4d6fa build-aux/Makefile.am: Accommodate step-level packages from proguic
Note: this really belongs in liza-proguic, and should be moved in the near
future.

liza-proguic is being modified to generate step-level packages, which are
significantly faster to build than larger ones (XSLT TAME scales
terribly).  These changes handle those new dependencies.

One important thing to note with this change is that suppliers.mk now
requires proguic to have run before generation so that those generated
dependencies can be properly examined.  This is a quick operation, so that
is not problematic.

This also depends on the .version.xml change that was previously made: when
the timestamp changed every time, we got into an infinite build loop.
2021-02-23 10:44:50 -05:00
Mike Gerwitz 80a61986bd build-aux/m4/calcdsl.m4: Do not generate suppliers.mk
This will be generated automatically by the Makefile.  It's not appropriate
to generate in the configure script, and I do not recall why I did
so---possibly to work around the issue of delayed tab completion when it
needs regeneration?

This removes suppmk-gen in favor of more generic Makefile targets---in this
case, having `%.tdat` depend upon `rater/core/tdat.xml`, even though that's
not quite true (the %.xml file generated from it needs it).  But these files
are going away soon; a pending TAME optimization branch removes support for
the underlying pattern primitive entirely; CSVMs should be used instead.
2021-02-23 10:43:09 -05:00
Mike Gerwitz 698ddcdd06 build-aux/Makefile.am (.version.xml): Only change timestamp on hash change
The timestamp of the file will now only be updated if the hash (version)
_actually_ changes.  This allows this to be used as a target dependency
without forcing a rebuild each and every time.
2021-02-23 10:41:20 -05:00
Joseph Frazer f7968c0513 [DEV-8362] Include program.mk in project root
If a "program.mk" exists in a project's root, it should be included in
the Makefile.

Co-Authored-By: Anthony Dalfonso <anthony.dalfonso@ryansg.com>
2020-10-01 09:58:45 -04:00
Mike Gerwitz da7a2c71c7 tamed: TAMED_JAVA_OPTS: New environment variable
This will be passed to dslc and then to the JVM.  The intent is to permit
fine-grained heap ratio tuning.
2020-08-19 10:19:04 -04:00
Mike Gerwitz 59b2b32756 build-aux/lsimports: Fix awk gensub warning
Third argument must be numeric indicating which match to replace.
This error did not exist in previous versions.
2020-07-23 14:31:54 -04:00
Mike Gerwitz 1fbcededa8 build-aux/Makefile.am (ui/Program.js): include-path with arbitrary parent
This handles moving to another repository structure (our gigarepo) where
this relative path is no longer true.  The absolute path generated by this
is okay since it's ephemeral and only used for this build invocation.
2020-07-07 16:41:35 -04:00
Joseph Frazer 09350d0ada [DEV-7504] Add make target for "graphml" 2020-05-13 08:04:48 -04:00
Mike Gerwitz b8d9128f18 POC for full graph 2020-05-13 00:44:13 -04:00
Mike Gerwitz 35010f6882 build-aux/release-check: Fix readonly variable issue 2020-04-29 15:38:22 -04:00
Mike Gerwitz 99f2d5054e Release notes and associated scripts
This begins providing release notes for changes and provides scripts to
facilitate this:

 - tools/mkrelease will update RELEASES.md and run some checks.
 - build-aux/release-check is intended for use in pipelines (e.g. see
   .gitlab-ci.yml) to verify that releases were done properly.
2020-04-29 15:33:46 -04:00
Joseph Frazer b52b5825e6 [DEV-7147] Build "xmli" files using "tamec"
Rather than copying the files, we want to start using "tamec" to make
the "xmli" files, even if right now all it does is copy the file.
2020-04-09 09:46:46 -04:00
Joseph Frazer f6bf042505 [DEV-7136] Add xmli files
Add a new step to the build process that copies the `xml` file to an
`xmli` file. Eventually, the new compiler will create the `xmli` file
and the old compiler will convert it to an `amle` file during the
transition.
2020-04-08 08:27:47 -04:00
Mike Gerwitz 8385b64e1d [DEV-7086] TAMER: Remove WIP linker warning
While it is true that this is still being finalized, the warnings originally
existed because tameld was not feature complete.  It is now.
2020-04-06 10:04:19 -04:00
Mike Gerwitz bfea768f89 Copyright year 2020 update 2020-03-06 11:05:18 -05:00
Joseph Frazer e613bd8a8c [DEV-7081] Add options to tameld
We want to add an option to set the output file to the linker so we do
not need to redirect output to awk any longer.

This also adds integration tests for tameld.
2020-03-06 09:41:55 -05:00
Mike Gerwitz 6939753ca0 TAMER: POC: Output xmle
This is a working proof-of-concept that will be finalized in future commits.
2020-02-26 10:49:00 -05:00
Mike Gerwitz 696a5ab371 build-aux/closure-externs.js: New file 2020-01-22 16:31:57 -05:00
Mike Gerwitz 61fe1af1cb build: Add revision files for xml{o,e}
This will force a rebuild and will be useful for upcoming changes.
2020-01-14 01:13:51 -05:00
Mike Gerwitz be296a241a build-aux/Makefile.am: Optional timestamping
Note that, because of the way this is implemented, the timestamps may become
mangled (multiple per line) for parallel builds.

Output can be prettied up in the future.
2020-01-02 10:42:08 -05:00
Mike Gerwitz c888e17e97 Revert "build-aux/Makefile.am: (program.expanded.xml): .version.xml dependency"
Now I remember why I didn't do this: it forces a rebuild of
program.expanded.xml every build.

This reverts commit 4f3dfc3bc7.
2019-05-07 14:04:11 -04:00
Mike Gerwitz 4f3dfc3bc7 build-aux/Makefile.am: (program.expanded.xml): .version.xml dependency 2019-05-07 12:03:30 -04:00
Mike Gerwitz f270220b11 build-aux/csvm2csv: Propagate csvm-expand exit status
csvm2csv was not failing when csvm-expand exited with a non-zero
status.  Further, the tests were written incorrectly to account for this.

* build-aux/csvm2csv: Set `pipefail' option.
* build-aux/test/test-csvm2csv: Fix tests.
2019-04-09 10:57:59 -04:00
Mike Gerwitz 4c61dfb1cc csvm: Permit all whitespace (including tabs)
While tabs aren't desirable, users that are not developers will be modifying
these files, and so we need to be permissive in what we want to
accept.  That doesn't mean that we need to forego occasional formatting, though.
2019-04-08 15:17:00 -04:00
Mike Gerwitz 1a35232bd8 Parallel build support
tamed was originally designed with support for parallel builds in mind, but
I hadn't completed that work because we didn't have enough hardware that
we'd benefit strongly from it.  That has since changed.

tamed will now spawn additional runners as needed to fulfill requests, which
works around the issue of not knowing how many jobs GNU Make is going to try
to do at once.

There were a couple minor dependency fixes/workarounds for now in the
Makefile, but otherwise everything appears to be working great.
2019-04-04 14:41:07 -04:00
Mike Gerwitz 7b7cf13607 build-aux/csvm-expand: {orig=>src} local arg typo fix
This does not affect its functionality.
2019-04-02 11:05:03 -04:00
Mike Gerwitz 9a1f916486 build-aux/csvm-expand: Spawn only one date and memoize
A table with a couple hundred thousand rows was taking minutes to
generate.  This gets it down to a few seconds.

* build-aux/csvm-expand (parse_date): New function.
  (parseline): use it.
2019-04-02 10:58:12 -04:00
Mike Gerwitz e022a3133d Copyright year simplification and update to Ryan Specialty Group
This now uses year ranges, which I'll update annually.

This also renames "R-T Specialty" to "Ryan Specialty Group".  The latter is
the parent company of the former.  I was originally employed under the
former when LoVullo Associates was purchased, by I now work for the parent
company.
2019-02-07 13:23:09 -05:00
Mike Gerwitz c7fec6a240 csv2xml: Import /rater/core{=>/base} directly
/rater/core is being removed.
2019-02-01 00:36:48 -05:00
Mike Gerwitz 017ca1f437 suppmk-gen: Properly exit with non-zero status on failure
* build-aux/m4/calcdsl.m4: Exit on suppmk-gen.
* build-aux/suppmk-gen: Exit on failure.
2019-01-23 16:04:01 -05:00
Mike Gerwitz fa378a654a Add lsimports and check-coupling
lsimports will be able to be used to replace the last remaining Ant script
that generates depfiles.

* build-aux/check-coupling:
* build-aux/lsimports: New files.
2018-12-19 14:20:24 -05:00
Mike Gerwitz 73f6b77771 [BC BREAK] check target supplier customization
This allows customizing from the command-line what suppliers should be
checked.  This motivation for this is both to run as part of a distributed
pipeline (where each supplier may be built individually), and for during
development of a single supplier.

BC BREAK: Note that this will now check for `package' in the test path for
UI tests.  To keep the old directory around, a symlink of `packages' to `ui'
would suffice.

* build-aux/Makefile (SUPPLIERS, suppliers_strip): New variables.
  (check-am): BC-BREAK: Build and check only requested suppliers.
* build-aux/progtest-runner: BC-BREAK: First argument is now test directory
    and all remaining arguments specify the supplier XML files to check.
2018-12-18 21:46:18 -05:00
Mike Gerwitz 32e3b16ec9 Makefile.am (program-ui): Remove standalones dep
We want to be able to build the UI independently of the
suppliers.  Historically, this did not provide much of a benefit, but this
change allows us to build independently as a job in a distributed pipeline,
and allows testing out the UI when rating is unneeded.

* build-aux/Makefile.am (program-ui): Remove `standalones'.
2018-12-18 20:58:29 -05:00
Mike Gerwitz f50b49542e Makefile.am (program-ui-immediate): Remove old target
This target has not been used for years.

* build-aux/Makefile.am (program-ui-immediate): Remove target.
  (program-ui): Use dependency of old `program-ui-immediate'.
  (.PHONY): Remove `program-ui-immediate'.
2018-12-18 20:53:58 -05:00
Mike Gerwitz 4105dc8fef Makefile.am: Add common target
* src/build-aux/Makefile.am (common): New target.
2018-12-18 16:40:35 -05:00
Mike Gerwitz 044498f03f Makefile.am: Copy srv/!(rater).js to destination paths
Note that such files may not actually exist, which is why `nullglob' is set
and the `for' loop is used.

* build-aux/Makefile.am (SHELL): Set `nullglob'.
  (program-data-copy, lvroot): Copy srv/!(rater).js to destination JS paths.
2018-12-10 10:51:03 -05:00
Mike Gerwitz 079d1dcfaf tamed: Do not stall if TAMED_SPAWER_PID is running
This will ensure that tamed does not stall while e.g. make is still
running.  This makes TAMED_STALL_SECONDS almost useless; maybe it'll be
removed in future versions.

* bin/tame (TAMED_SPAWNER_PID): Export variable.
* bin/tamed (TAMED_SPAWNER_PID): New variable, default to PPID.
  (spawner-dead): New function.
  (stall-monitor): Use it.
  (usage): Update documentation.
* build-aux/Makefile.am: Set TAMED_SPAWNER_PID to own id and export.
2018-12-03 16:25:25 -05:00
Mike Gerwitz cc7e09a700 Add new c1root and local c1-service copying to build
This maintains BC for existing raters that have not yet been migrated to use
the new c1-import service.

* build-aux/Makefile.am (path_c1root): New variable.
  (.PHONY): Add c1root target dependency.
  (program-data-copy): Copy to `@C1_IMPORT_MAPDEST@'.
  (c1root): New target.
* build-aux/m4/calcdsl.m4 (C1_IMPORT_MAPDEST): Configure depending on the
    existence of the `c1-import' directory.
2018-11-28 15:55:49 -05:00