Why we need to talk about Kiwis

The alternative narrative that I’ll champion here, is that a larger share of the Missing Million is actually Kiwis, probably relatively young ones, who have come to Australia to work or study, but spend rather significant amount of time (in various trip-lengths) back in New Zealand. I won’t try to prove that it’s the dominant component, since there isn’t clear evidence for that either, in fact it appears that the Missing Million must comprise quite a variety of traveller types. But it fits the data better than the Boomers in Bali story since departures to New Zealand are the dominant component of short-term departures, so it’s a reasonable angle to take to refute my ‘rebuke’.

It’s also a fun narrative to choose because it contrasts so well in every sense with the Cameron’s hypothesis in a policy sense. Unlike Aussie Boomers returning from their retirement holidays, it’s quite possible that the New Zealanders will actually make a general exodus at some point, and decide that New Zealand is at least as good a place to live in their next life stage, such as raising a family or retiring. It’s also possible that they might continue to use a significant number of services (like health-care) in New Zealand on their trips back home.

Perhaps best of all, the case of the Kiwis highlights the misnomer that ‘Net Overseas Migration’ being closely linked to “Permanent Migration Program”, which small-Australia advocates like Macrobusiness continually urge the Federal Government to reduce. Unless we want to tear up our reciprocal arrangement with our New Zealand brothers over the ditch, this significant component of migration isn’t something that cutting the permanent migration program will have even the slightest impact on. Kiwis don’t need a permanent visa to live here permanently, and neither do we in New Zealand.

As we can see here from the Department of Immigration and Border Protection, there are more New Zealanders in Australia than any other type of temporary entrant. Well over 600,000.

Figure 2: New Zealanders make up the largest share of ‘Temporary Entrants’ in Australia

However New Zealanders have reciprocal arrangements with Australia that allows them to remain ‘temporary’ permanently, and do all the things that we normally associate with permanent residents, like working, and coming and going as they please. As a result, they actually constitute a surprising slab of the official “Net Overseas Migration” intake. In fact, since 2008 when Australia’s supposedly turbo-charged migration intake really started, New Zealanders have contributed just as much as almost any other category except students.

I can’t help but make further passing observation here, that all the permanent visa categories (in black boxes) during this period rarely add up much past 70,000, which is about the number that most of the small-Australia advocates advocate anyway. It’s the influx of temporary entrants that has really driven the appearance of extremely high migration during this period, which is consistent with the broad thesis I outlined in a previous post about ‘mobility’ rather than ‘migration’ having made a level-shift. Eliminating the entire “Permanent Migration Program” from 190,000 to zero won’t do anything to reduce two thirds of the migration we actually have. In fact it would probably just drive a further increase in temporary entrants, including on Bridging visas, but I’ll focus on that charge another day.

The Boomers in Bali Narrative

The image Cameron Murray created of the Missing Million relied on four characteristics in short-term departures: reason for departure (holiday), length of stay (short), destination (Indonesia), and age (older).

Sure enough, he has a chart which says the highest level of growth comes from growth in holidays:

Figure 4: Holiday travel seems to have risen the fastest

And of course, there’s a graph which says that the fastest growth is in short-term trips as well.

Figure 5: Very short travel seems to have risen the fastest

And departures to Indonesia also seem to have risen the most sharply from the mid 2000s as well:

Figure 6: Indonesia rose very quickly as a destination

And there’s a hint as well that somewhat older people might make up a higher fraction of travellers of lately too.

Figure 7: Older people are travelling more in relative terms

So it would be very tempting to conclude that the same set of travellers are driving the relevant trends in all of those cases, and that those travellers also constitute a significant share of the “Missing Million”. To do so would be to essentially make a whole sweeping set of assumptions which haven’t even been discussed, let alone tested, and some can be demonstrated to be substantially untrue.

Perhaps the easiest to tackle is the last graph, whch was used to suggest that a significant number of travelers might be the old boomers. The problem with this graph is that it’s expressed as a percentage. As I’ve previously outlined, the most striking trend in overseas travel is that it’s increased, quickly and relentlessly. If the two curves shown in Figure 7 referred to numbers of movements, the 2015-16 lines would be consistently far higher (about one and a half times) as the 2005-2006 lines. Whilst a somewhat higher share recently are 60-something than previously, movements are now so high in all other categories that change that caused the discrepancy could be in any other age-group just as easily.

With regards to the other data, first and favourite technique to test a hypothesis is to just have a look at relevant data with the Mk 1 eyeball. Adding in the cumulative discrepancy between NOM and Net Movements, we can see how and when the Missing Million actually grows:

As we can clearly see there, the discrepancy between NOM and Net Movements seemed to have really become serious in the very early 2000s, quite possibly before the marked acceleration in Short-Term Departures. It does however appear that Short-Term Departures do seem to track somewhat with the cumulative discrepancy thereafter. However, the same isn’t the case with Departures by destination:

Here it can be see that the accumulation of the first half of the Missing Million occurred while travel to Indonesia was actually low, and not growing. Furthermore, at precisely the time of the fastest acceleration in travel to Indonesia, NOM and Net Movements came closest to aligning. In contrast, travel to New Zealand grew consistently throughout this period, not to mention being higher throughout. I’d argue that NZ looks more likely by far at this stage.

My second favourite technique is to do just to a quick of some numbers, to see whether things add-up even roughly, at least to within an order-of-magnitude. Looking at the Short-Term-Departures by duration, we can multiply each series by an appropriate number to see what sort of impact each component might actually have on people actually absent from Australia. Not having any better information, I assumed that the distribution of movement lengths in each category was reasonably flat, and chose 4, 11, 22, 46, 77, 165, and 273 as the probable average number of person-days that each departure would reduce from Australia’s Physically Present Population. I then took the previous 12 month calculation and divided by 365 to find the total number of ‘person-years’ that each set of departures could contribute at any given point, to see which contributed the most.

We can see here that the impact of trips less than two weeks is very greatly diminished. The largest contributor to the Missing Million is likely to be trips of over a month, hardly a fleeting holiday.

But more interestingly, we can see that the short-term departures of all the trips under two months clearly can’t add up to anything like a million people. I did some calculations and found that in fact at the end of the data all the short-term departures probably only cumulatively add up to under 800,000. I suspect the assumption of a flat distribution across all the categories is likely to overstate absence if anything. Furthermore, at the end of 2001, before the Missing Million had left, those departures accounted for about 400,000 people absent. So it really isn’t plausible that all these departures listed actually account for anything like the full Missing Million, or probably even half of that. The sub-set of Boomers that Cameron Murray describes is doomed to be a trivial minority in any case.

There’s also the possibility of significant a shift internally within categories. It could be the case that my estimation of a flat distribution across each time interval is false, and some change has led quite systematically to some of the categories to become increasingly skewed, probably to the left (shorter trips) since the frequency of travel has risen so much. The switching over of the 6-12 month and 3-6 month lines seems to show some evidence of that occurring, however if a shift in that direction also occurs internally within the categories, we would likely have even less of the Missing Million accounted for.

A reality-check on Data Quality

There are a few things that could explain such a result, and as always it’s probably best to go back to the source of the data to try to understand why it is possible. A significant factor is that the data presented here is based on peoples indicated intent, as per their passenger departure cards, like this one:

Figure 11: The departure card assumed that people know whether they are ‘resident’ or ‘termporary entrant’.

The largest failure in these cards is asking people to self-select what type of traveller they are, most confusingly whether they are a ‘visitor or temporary entrant’ or ‘Australian resident’. Since these cards don’t come with any explanation of the 12/16 rule of ‘residence’ for migration purposes, these cards would make no sense for our increasingly part-time population, many of whom are foreign citizens, here on a temporary visa, but staying (as a Student, worker, or backpacker) for long enough to officially qualify to be ‘resident’ for migration purposes. So the entire data-series we’re working with here really is likely to be fraught with uncertainty, and doesn’t promise to consistently include those who should be counted, or exclude those who should not be.

Furthermore, there’s absolutely no obligation on the traveller to honour their ‘intent’ regarding travel. Often they may not have actually booked their onward or return flight. If I had to speculate a little as to the micro-economic dynamics that might be at work, I would think that plenty of travellers (particularly those on temporary visas in Australia) would wind up indicating something that was a poor reflection of what actually happens. With more and more people making multiple stops on their travels, with fewer of them planned in advance, they’re likely to make some educated guess about what the scary people at the customs gate are going to be most happy to hear, (including about their residency status) and just report that.

To add to this, not all the data captured in departure cards is comprehensively enumerated. According to the ABS, on average, only about 5% of the cards are selected for a sample, and most of the rest carefully imputed. Many of the methods I’m using below could well struggle if there’s even moderate errors or poor assumptions made in this sampling or imputation process. The strange results produced by our New Zealand regression below could be an indication of uncertainty or inconsistency in this process as much as anything else.

But perhaps more importantly, given the way that migration is defined under the 12/16 rule, there’s absolutely nothing preventing these ‘short term’ travels actually contributing to Net Overseas Migration, and hence not explaining a discrepancy between that number and Net Movements. Two six-month trips a couple of months apart will constitute migration. As will a two-week trip, if some was travelling a lot in the previous year.

Missing arrivals instead?

Trying not to be disheartened, we could look to see whether what data we do have could still clarify things further. So far we’ve only looked at half (or less) of the story. We have short-term arrivals as well:

Here a couple of things are striking. In particular, the under-1-week movement trend is ususally the highest, followed by 1-2 weeks. People arriving in Australia seem to report a far shorter intended stay than those leaving. And, the total number of arrivals is far far lower than the departures. This is also consistent with the hypothesis I’ve outlined earlier, that we’re a hard pace to get to for a short trip, but a good place to leave from for one.

Here the plausible ‘second-half’ of the story emerges. By far the largest contributor to ‘person-years’ present in Australia from short-term arrivals is actually from the longest intended trips, of over six months, and the second-longest from 3-6 month visits. Importantly, there’s a far larger possibility that these ‘short-term’ visits will wind up being counted in Net Overseas Migration. Students or backpackers who stack up a couple of 6+ month visits inside a couple of years (probably even more likely than staying continuously) will almost certainly officially ‘migrate’. In stark contrast to the case for departures, the shorter categories are relatively tiny, and didn’t grow at all during the period when the missing million emerged. This goes a lot further towards explaining the true origins of the Missing-Million, and is further supports the hypothesis outline in my earlier post. The sorts of ‘visitors’ we get tend to be longer-term visitors, but still visitors, where as our travel outwards tends to be for faster visits.

Again, summing the total ‘person-years’ accounted for by this trend, we find there are just under 800,000 people in 2017, and just over 400,000 in 2001. Three quarters of this increase can be accounted for by movements where the stated intent longer than six months. If a large fraction of these are counted within migration, and I think it’s almost certain that they are, then the person-years present due to short term visitors would barely have moved. If a significant share of the 3-6 month intentions were also counted as ‘migration’, then the person-years present would have moved backwards due to short-term arrivals. This could explain a further part of the discrepancy between Net Overseas Migration. However, it also seems likely to not add up to be enough to reach the million.

To confirm the failing significance of passengers’ stated intentions, it looks like the ABS is also going to abandon these particular series.

An interim summary

So far, having just eyeballed some data and done a couple of quick calculations to see how the numbers stack up, it appears:

1. The Missing Million grew to half its height while travel to Indonesia was low and flat, and travel to New Zealand was increasing.
   
2. Person-year absent due to ‘short-term’ departures is driven by travel between two-weeks and two months. Very short holidays don’t contribute much. But all this data on stated intent can’t account for the Missing Million.
3. Person-years present due to arrivals is driven by long stays of 3+ months, and overwhelmingly by 6+ months. If a large slice of these are counted in ‘Net Overseas Migration’ movements, the loss of arrivals could account for another part of the Missing Million.
4. The data is fraught with uncertainty in any case, and all our work here should be taken with a grain of salt.

Unpacking correlation

What we haven’t yet tackled seriously is the assumption of a high-overlap between the four different trends, or at least the three which we have data on. To establish or refute this, it’s better that we look at the information available to us, in its raw (monthly) format. As we can see, the intense seasonality present in some of these time-series can provide us with a key to possible analytic mechanism that could investigate the plausible degree of overlap.

(Note, one can click the countries in the legend to turn off a series, or double-click a country to show a single series. This helps to isolate or inspect individual trends.)

From the above two charts it can be quite easily seen that the slump in departures growth that occurred in the mid-2000s and the dramatic elevation in it after about 2008, is concentrated in South East Asia and to some extent the USA. This is again consistent with Australia now competing with other attractive pacific destinations as holiday spots, but probably acting as something of a home-base for people wanting to explore South East Asia. Arrive long-term to Australia, but leave short-term, frequently, to explore Asia.

It should be noted here that while holidays have been the largest growing single component, there’s clear evidence here that visiting friends is a significant driver of the December annual peak in departures. This seems also to align with a strong December Peak in departures that occurs in departures to New Zealand (which is less present in departures to Indonesia), and also in slightly longer travel times, including 1-2 month visits.

All this strongly points towards some significant overlap between the largest contributors to ‘persons absent’ due to departures being New Zealanders visiting friends or family back home, rather than boomers in Bali, though the degree of the overlap is still hard to prove.

Dive with me baby!!

We could go further to try to substantiate that claim using some more advanced statistical techniques, for those who want to dive even deeper into the data. For example, we could use a decomposing function to isolate the seasonal and trend components in a given time series, such as this call of stl() does this using the loess smoother:

Figure 17: A decomposition of departures for the purpose of Visiting Friends or Family

We could then compare the seasonal component (in the second pane) with other seasonal structures in different category types. However, as can be seen from the asymmetric errors in the bottom pain, the growth in the seasonality of the series isn’t perfectly accounted for. This highlights that the structure of the seasonality is subject to change as well within the time-series. Again, my preferred method to get a sense of how this might be the case is to use the Mk 1 eyeball again for a quick check. Another visual representation of the raw data is to overlay the different years on top of one another.

And one can see intuitively how there could be a significant overlap with this sort of series:

And also might have quite a strong overlap with this sort of series:

Or at least, one could say it probably has a stronger overlap than with this sort of series:

Which might have more of an overlap with this sort of series:

All in all, this really just serves to pose more properly the question: What is the actual composition of these time series, in terms of the other ones? Are departures to Indonesia predominantly short, and departures to New Zealand substantially longer?

Dive Deeper

It might be possible to attain to substantiate this sort of eye-balling activity statistically using a multivariate regression. Essentially all this is doing is seeing which combinations of one division of departures corresponds best to a particular series in another characterisation. This is easily done, though care should be taken when interpreting the results, a summary of which might look like this:

## 
## Call:
## lm(formula = Numb.of.move.New.Zeal ~ . - Date, data = kiwi_length_regressor)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -41967  -6259   -269   6877  30353 
## 
## Coefficients:
##                                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                     1.381e+04  3.119e+03   4.427 1.33e-05 ***
## Num.of.mov.Und.1.wee            4.748e-01  6.952e-02   6.830 4.49e-11 ***
## Num.of.mov.1.and.und.2.wee     -1.661e-01  5.369e-02  -3.094  0.00216 ** 
## Num.of.mov.2.wee.and.und.1.mon  3.578e-01  5.254e-02   6.810 5.09e-11 ***
## Num.of.mov.1.and.und.2.mon     -6.041e-02  5.835e-02  -1.035  0.30132    
## Num.of.mov.2.and.und.3.mon      1.046e+00  1.269e-01   8.248 4.65e-15 ***
## Num.of.mov.3.and.und.6.mon     -2.139e+00  1.677e-01 -12.752  < 2e-16 ***
## Num.of.mov.6.and.und.12.mon     1.648e+00  2.061e-01   7.995 2.58e-14 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 10920 on 310 degrees of freedom
## Multiple R-squared:  0.8832, Adjusted R-squared:  0.8806 
## F-statistic: 334.9 on 7 and 310 DF,  p-value: < 2.2e-16

Probably an easier way to understand the output is to say that a model was constructed attempting to reproduce the data of Departures to New Zealand, by adding different amounts of all the series for different durations of departure. The model that was fitted to the data ended up looking like this:

And was composed of the following weightings of the length of departure series:

Uncon Kiwi Coefs

This is shows something odd, it appears that there’s a negative weighting of several of a couple of the components. This doesn’t make any sense, as you can’t have a negative number of departures. In addition I’ve shaded the colours here with the log of a variable of the output of the model called Pr(>|t|), which gives an indication of the likelihood that of this original data occurring if that co-efficient wasn’t present at all. In other words, darker colours give a higher degree of confidence that that particular coefficient weighting is likely to be good.

Seeing relatively light colours for slightly negative coefficient values is probably ok, (as it’s quite probable that having none of this component, or a very small positive value wouldn’t throw out the fit too much) but finding an extremely dark one for a large negative value means that the results are overall likely to be somewhat unrealistic. It appears that the negative value is likely compensating for over-weighting of the other two components to the left and right. To understand why, let’s look again at what the spectrums suggest:

Here we can see why the regression came up with this result. Departures to New Zealand slump in May each year (see Figure 19), and 3-6 month departures have a peak in May. Subtracting that component consequently improves the fit substantially, though it’s not a physically possible result. Overall, it appears that my attempt to get these statistical methods to substantiate a proposed split between the different visit duration seems to have failed. It’s probable that in New Zealand’s case the implicit assumption (that the distribution of trip-lengths to New Zealand is relatively stable over time) doesn’t hold well. We’ll relax this assumption a little later to see if it improves things, but first let’s just check to see how close a more-realistic model can come.

To get a more realistic model, we can re-run the regression with a constraint that requires all the co-efficients to be non-negative. Happily there’s an R package called nnls by Katharine Mullen which can quickly create such a model. The result is this:

Kiwi Constrained

This produces two curious results. It shows that with a constrained model not nearly as much of the seasonal fluctuation can be incorporated, and the December spikes in particular don’t match up as well. However, in an odd twist of fate, the fit of the 12-month running average is actually better, with a slightly higher R-squared match. This suggests that the adjustment made to the coefficients between the unconstrained model and this one are almost certainly more in line with long-term, as opposed to intra-year trends. By plotting the co-efficients, we can see what that is:

Con Kiwi Coefs

Not being able to include a large negative weighting means that the magnitude of the other positive weightings also decreases. In this case, I strongly suspect that the lower weighting of 6-12 month departures (which were completely flat in overall trend) actually assisted in better matching the trend. Sadly nnls() doesn’t yet have as easy a way of pulling out Pr(>|t|) so I wasn’t able to give that indication of confidence with a colour scaling.

The graph also shows that we have a weighting of 6-12 months that is over one, which is also unrealistic, since we can’t have 120% of 6-12 month departures all going to New Zealand. With more time and resources there would certainly be means of further constraining the models, including with the outputs of other regressions on other variables, to make a fuller and more consistent model. But first let’s just multiply these ‘weightings’ by the mean actual number of travel undertaken under each departure length, and then by the average length of stay, to convert the coefficients back into numbers which are easier to make some sense of: numbers of departures monthly, and number of ‘person-years’ absent that could be attributable to that flow.

This leaves us in a somewhat unsatisfying place. It indicates that there might be a significant number of people ‘absent’ in New Zealand due to longer trips, but we also know that it’s impossible that more than 100% of the trips over 6 months went to New Zealand. The model isn’t producing realistic values yet, so we can’t have any real confidence in the levels indicated there. In addition, these values are taken for a mean for the entire period, which isn’t actually that helpful since we’re actually most interested in how the trends look over time.