Dyalog ’11

Barry Crown is a world expert on traffic circles ("roundabouts" in the UK). Like many domain experts, he has written a little bit of very clever APL code to help him along, which he then sold to traffic engineers to help them along too. Unfortunately it is coded in APL*PLUS/DOS and it just stopped working. This is the story of the rescue process, spiced with some of Barry's wilder stories from the history of roundabouts, good, bad and ugly.

In the business world where APL is being used to perform large scale data processing, data matrices tend to be very large, with millions of rows and thousands of columns of data. In many cases, the data matrix is so large that we can only fit a small slice of it in our computer memory. When you work with a very large matrix inside a loop that consists of millions of iterations, optimising matrix indexing is important in reducing the run time of functions. There are many cases where an application program running for over 24 hours can be reduced to 12 hours or 6 hours run time when the data indexing has been optimised. This presentation introduces several techniques that can be used to optimise APL matrix indexing based on the presenter's experience with very large data matrices and spreadsheets.

For many years we have been talking about how APL is perceived by our prospective clients and what is expected when new systems are delivered. Today, the expectation is much higher than ever before, which in turn offers us new challenges and opportunities. This non-technical discussion looks at APL from the perspective of three people; unfortunately our client is unable to attend the conference so instead Paul Grosvenor will explain his situation from the point of view of someone with many years of experience in IT but no in-depth knowledge of APL. His company merged with another who already had products written in APL, and as a result he was forced to adopt a number of systems which he had little understanding of, nor desire to continue with, in their current form. His position is one we may all be familiar with but the outcome may be very different...

Our second view is from Simon Fitall, CEO and co-founder of Galileo Analytics. He has been using PC APL for many years in a number of companies to great effect. He will detail why APL has worked for him, why he continues to make use of it and why he is happy to develop new systems with it. Our final view is from Anna McCollister-Slipp, Chief Marketing and Communications Officer of Galileo Analytics. She has no technical knowledge of APL or wish to gain any but her primary interest is making her products look different. Today, our customers expect a much richer and better designed interface than before and usually they are well versed in the use of GUI to best effect. Anna's challenge is to make a "standard" [APL] application stand out from the crowd, to be just a little different and gain that essential first look wow factor.

Finally, Simon will take us on a very quick tour of one of our recent applications, COSMOS, which fuses together Flash and APL to produce a new, web delivered, application that looks and works just a little bit differently.

Optima has spent the past 12 months or so developing a new Clinical Data analysis system called COSMOS which is just about to go into production and be used in anger for the first time. It is being discussed briefly in "APL from 3 sides of the fence." COSMOS is a brand new product for us and it is designed to allow Researchers, Practitioners, Pharmaceutical companies, etc. to view their clinical data in a more organic way. Our system makes no assumptions on what the data means other than what is necessary to be able to display it. COSMOS, therefore, presents an unbiased view of the information and, by using as much data as it can, provides a more complete picture of what is going on in the real world.

The aim was to be able to show multi-variant, time series, clinical data on screen in an intuitive way and allow the user to identify data "hot spots" both quickly and easily. We hope to show how the design has developed over the year, what we were trying to achieve and how some of the code modules work together to produce the final result.

It takes more than three chords and an old guitar to become a country music star, and it takes more than a workspace full of code to produce a professional APL application. This presentation will cover some of the issues that arise when making the journey from coder to developer, with a particular focus on automated testing, including the automated testing of GUI code.

The Windows COM (Component Object Model) interface is still useful to tie everyday applications together with APL. At Montreal's BCA Research, the primary APL application produces charts for global investment research. Dyalog's native ?W... functionality has been successful for its ease of customisation and flexibility – thus far – in delivering these charting results and analysis to internal staff and customers, but the need for integration with the dominant office and online platforms continues to drive innovation.

Hence, based on a suite of utility functions for APL and Excel integration developed by David Crossley, two new tools are emerging for BCA researchers to deliver charting functionality beyond the APL workspace. The "XLCHART" function manipulates the Excel object to chart timeseries data – according to "BCA standard" features. The same charting code was then used to create "BCAXL", an Excel spreadsheet that uses the APL OLE server object to retrieve and chart BCA timeseries data, using VBA. The Crossley toolkit, limitations, lessons learned and future goals will be discussed.

SimCorp Dimension is an integrated single database investment management system mainly based on Dyalog APL, partly C#. As our customers grow in size, so do the data volumes handled in the system. To meet the request for handling larger data volumes, going 64-bit is the obvious and easy choice due to the increased address space. Stig will take you through the steps needed, and the pitfalls to be aware of, to get successfully through the migration process and finally release the product. The main challenge was actually the explosion in memory usage, but also to get all the interfaces to (for example) external C libraries and third-party products changed.

Vstar has similarities with MySQL, Oracle, DB2, kdb, etc. but has differences that might interest you. It is implemented in J and is open source. It has been under development by Vectornova for more than a decade and has evolved through serious production use and is now in its 4th generation. Although not targeted in any sense at APL applications, the architecture and mindset might have particular appeal to APL developers. Although all client types are supported, the natural fit with APL clients can be rewarding. Vstar has SQL, but real power comes from non-SQL based on decades of APL and J array language development. The presentation will be informal and focused on architecture and design goals, rather than confusing demos of blistering performance.

Phil has had a long interest in extending the language with user defined operators, thinks too little use is made of them and believes that many APLers are a bit afraid of them.

He will develop one or two of them before your eyes and perhaps demonstrate one or two more. At least one of them is actually quite useful and he cherishes a hope that one day it will be implemented as a primitive.

Most of the research into creation of high-performance code for compiled array languages has focused on optimisations for array operations, including such methods as loop fusion and array contraction, with-loop folding, with-loop fusion and loop peeling.

Such optimisations can improve the performance of array-based applications by increasing the level of available parallelism, by reducing memory subsystem traffic, by increasing locality of reference to array elements and by amortizing loop overheads over several related computations. However, it turns out that, for many applications, small arrays, such as index vectors, are a curse, because of the overheads associated with their creation, initialisation and destruction. When such arrays are materialised in the inner loops of computationally intensive code, substantial performance loss – up to two orders of magnitude – can result. Our recent work attempts to address this problem by rigorous scalarisation of index vectors. Our preliminary work shows significant performance improvements in many of the APEX benchmarks as a result of such scalarisation. Our talk will show why scalarisation is needed in practical applications, will describe scalarisation and will present our current research results.