Episode 156 - Server Latency Debrief and Wayfarer gets how many nominations now???

Hello agents, welcome to the Ingress Insights Podcast. Hosted by Pokémon Trainer G.A. Ingress Insights aims to give you the latest news and updates about the Niantic Spatial Game Ingress, whether you're a veteran agent who has recursts 17 times or a brand new agent working to level 8, wondering what a fracker is. This is your place for the latest Ingress News and Discussions. In 2020, C's Ingress renewed under Niantic Spatial and many new campaigns to engage agents. So, plenty to discuss. Share any thoughts by checking our social media links now. Over to Ada. Additional Intel Available Hello agents, I'm Oak2F1156 of the Ingress Insights Podcast. This week's episode is beginning over an apology from the Ingress team for the anomaly mess last weekend. The start of the courier files and also be looking ahead at what's happening in the game in June. We already have this schedule for June. Before we get into all of that, I do want to go through my own progress on Ingress. So, as you might know, last time you listened to I was getting close to level 15. Sorry, level 16. I was level 15. Level 16 is currently the highest level in Ingress. If you want to progress further than that, you need to recurse which research your level back to one. And then you start to go all over again. It's easier to level up to your original all the badges and you get a few extra little perks for it on there. So, I recurced at the weekend. I think it was on Sunday or early Monday that I actually was able to hit the button. Now back up to level 6. So, I'm on 221,690. Sorry. 221,698 out of 300,000 AP that I need to get to level 7. The higher level weapons of the game are level 8. So, I mean, I'm just lacking some level 6. I think you have some stuff for the level above. So, some level 7 stuff as well. So, I'm waiting to get to level 8 again and be nice to use the burst just and use the power cubes at the right level on there. So, I got quite used to that, but I leveled it pretty quickly. I think I went from like 1 to 5 in the afternoon. It's just a little bit 6 to 7 going on there. And then 7 to 8 shouldn't take too long either. I'll be back up to level 16 again pretty soon, I'm sure. With that passed looked out the way, I go into the anomaly news for Orion and start off with the apology from the ingress team and details of what they're doing to sort it out. So, I covered about the latency issues during the anomalous during the May 16th weekend. Since then, several, I guess, ingress news channels updated and shared the similar message. It reminded me a bit of the Pokemon Go one, they shared. And there was the letter for it, which I don't even remember the name of now, but that campaign just to try and get the ingress team to listen a bit on there. So, it's shared by like ingress updates English, I assume that other language channels as well, the EGR and RGNN following this Brian posted an update on the ingress news site on the 22nd of May saying May 16th server latency debrief on today May 16th agents experience severe server latency, game at consent to register in general scan it instability. So, we know this is not isolated to Orion, Sydney and Orion, Praga, and Nomale live events, the ripple effect of photo player's worldwide. As you may know, an XM anomaly generates a hyper-constrained spike in database action at smaller reporters, hundreds of agents deploy, attack and mod the same portal that's the same millisecond, ingress of and running on a new database since before the plus gamma, high to bad and plus gamma, when is there is anomalous, but Orion was the first anomalous is with this increased level of agent activity. This past weekend we experienced a perfect sum of three major bottlenecks, the first was right contention gridlock and new database handles simultaneous updates attempts differently than our previous database. Above a certain level of traffic, the retry attempt starts at number the first try attempts and the ability to make updates slows and dramatically. The simple one was tens of thousands of ghost records, game objects like portals and reddened is changed constantly, so our database utilizes an index to track each change. However, every change leaves find a ghost or temporary tombstone road that stays in this release one hour. Our database becomes clogged with ghosts and delays were caused by needing 60 tens of thousands of ghost records to find the real game actions. Because our database has a global data, this slowed the global portal network, creating the ripple effect latency experience like this outside the anomaly zone. Three slow automatic scaling, we use auto skays, all systems auto at the add more computers to the cluster when there are more requests. The start of anomaly comes with a sudden rush of traffic which the auto scares cannot respond to quickly enough, so it preemptively forced more passing systems before the start of each anomaly. However, in combination with the issues above, the auto scares still could not keep up. While team live monitors and scores every anomaly and we have additional team members on call for when it's happened, when our automated alerts identified issues and page team members manually bypassed our automated systems and forced service onto hiring machines, trying to handle the processing load, overscaled our database instances far beyond our automated limits and forced background systems to stay at max capacity so they would not drop further connections. These brute force updates didn't grow from crashing but they did not fix the architectural issues we were seeing. It's time to prevent these traffic jams and ghosts from disrupting future events, we're currently working on the following. The first point is to move to in memory portal updates, we're completing a new game entity system that should allow game actions processing from step temporary server memory first. Buses the ghosts, we enter a new ghost but we are changing how database index is time, this should eliminate major what we're going to identify last weekend during the anomalies and the third one they're trying is a pre split anomaly zones. Since we know how each geographic arose will receive x traffic during anomaly, we'll split these rows into their own shards so the database can allocate resource to those sections of data instead of having to simply active areas and then reorganize the data after the anomalies already started. As an additional mitigation step, we are going to temporarily reduce the xMP attack radius because of the right content we're seeing with simultaneous updates when xMP is fired. We will increase xMP attack strength but reduce the attack radius so that fewer ports are attempt by each xMP fired. This temporary address will go live Tuesday 26th of May 5 PM ETC and revert back to its current attack radius on Monday June 1st at 5 PM ETC. We apologize for these issues and I hope you have a clear understanding of what happened and what we are doing to address it. We are working with Orion anomaly points of contact to distribute Orion anomaly male passcode agents who contribute to the Orion Sun Sydney and Orion Prague anomalies and need a passcode and we're also discussing a make good specifically for Orion Sydney and Orion Prague anomaly participants. Additionally we are adding a make good to store free item category available to all agents which change the following. 1A is shield, 4R multi hat, 20 power cube, 20 xMP, 20 resonator, 8 rare heatsink, 1A to refractor, 1 Jarvis virus, 1 Kinetic capsule, 4 very rare shields, 8 rare shields and what's off bank ultra link. This free load-acquit will be available 7 days starting Friday May 22nd 5 PM ETC to Friday May 29th 5 PM ETC. Thank you for your patience, your detailed book reports and your continued dedication to ingress. As always we are committed to continuing to prove ingress and thank you for your support. Orion on behalf of ingress team. So the main takeaway there is that they know there are issues and trying to do ways to sort it out but obviously we don't know if it will work. They've got to try things. The interesting one is reducing the xMP attack radius because that's only for a little while they're doing that from May 26th to Monday June 1st. That does cover the anomaly period we've got coming up next weekend in Cure City and Jersey City. So we interest see how that affects that anomaly. Additionally there is the make good bundle available free in the store. It's got quite a lot of good stuff in there. I won't claim myself until the 29th possible because I'll try and get back to level 8 so the xMPs and the power cubes are level 8 ones on there. But it just seemed like the ingress team at least listen to know that there's some ways they can do to fix it and they're going to try it and we just have to find out after the Cure City and Jersey anomalies next weekend to whether it worked basically. There is a new model that's been added to the store as well so I'll go through and discuss that next as part of the Orion anomaly series a new bundle has been added to the store as well with a badge in so this is the Victor Careys 2026 bundle showing Victor sharing the sky with their spatial background behind them. It features the Victor Careys 2026 medal and 5 very rare portal shields for 30,800 CMU. Similar in design to the Cureys Saga 2026 bundle we've got already. I believe there's another one yet to come as well on now before the end of this anomaly season wraps up everything related to the Orion anomaly so I'm moving to some general ingress news next. We'll start with the smallest bit of ingress news which is there was a new update released a few days ago so this is version 3.9.2. It says sure tackles a young referencing the agent name that everyone saw on their completed anomaly bounties but showing he played a normally imminent campaign is not we fix the issue causing this campaign to up for all agents also. fix the red dot notification pick on drone net in the main menu so it was on that wouldn't go away reminding quite a few people of the Harry Potter was the unite with the red dot that stayed there so those are gone and sorted in the latest updates to make sure you're all dated on there we have also had the choreo files start today on the 22nd of May so you'll be able to see that campaign in your scanner on there and you'll see the tasks as I've run through these before but the first ones are Ernie 90 glyphac points deploying 25 port mods destroying 42 port mods and recharging port with 250,000 xm doing those will get you well doing any three of those will get you 400 plus sorry 400 of Ryan tokens I'm so confused than it all called plus something as well as three level eight ultra-trikes and one media so it seems like the media is being used again for a bit of lore on there so that's nice to see in this one we do still have all the Nemesis Redux campaigns going on for a few more days so make sure you complete those on there if you have picked up the other ones and then finally we've got the periodot keeper campaign coming up on the first of June so that only now be able to see in your scanner starting soon it just says thank you for raising your dot I've been through the task list for that one as well and you get a medal for that so it's always nice to have an extra badge on there and as a reminder for periodot mobile as well that is about it for general ingress news I guess the main bit was the anomaly apology and the choreo file starting off so we'll see how those are progressing on and we've got the anomaly next weekend as well so that'd be interesting to go through I've got a little bit of way fair as such recon stuff to go through next next up you actually get an ad break hello agents hope you're happily hacking away whilst interrupt this podcast I'd want to talk for a second about the tea public store that I run for ingress insights which is over at tpb.com/user/ingress-insights I have been putting up quite a lot of new designs like you be which can get on socks shirts mugs tote bags to name just a few things so for may have added quite a few designs on there some of us already so there's one which is a kinetic capsules shirt doing an image of two kinetic capsules saying I don't want to be outside but these kinetic capsules aren't going to walk themselves which I've seen already self one with a quantum capsule on with like a blurring on it to show that echo sore main of the quantum capsule there's also one featuring a hyper cube which I was watching a certain film and then sort of video on there says phenomenal cosmic heart e-tpt cube for the hyper cube and then the final two new ones that I've uploaded this month is one with a heatsink saying why would you only hack portal once and then finally one saying only keys showing the speedcliff on there because obviously you only get keys with that speedcliff anytime new items get ads he will restore they are discounted for two days so you can get them for example instead of 19 pounds it's 13 pound for a shirt again there's discounts on wherever you're getting on there for those two days because of brand new design so I would ask that you do check it out from time to time if there's design that grabs you on there or if you think there's something that is changing on design do let me know finally I'm trying to increase the visibility of the podcast on Apple podcasts so if you do listen on that service you have a few moments spare I would appreciate if you rate it and then leave me any feedback on there that's it for this advert I'll let you go back to your regular scheduled podcast now before I do get into weigh fair and recon I'm not sure what I'm doing next week for the podcast I may be a bit tired after my break so just in case I'm going to run through the events we've got coming up for June I already mentioned that we've got the periodot campaign coming on the first of June so that'd be interesting to do and get the mail from there then on the sixth of June we've got a first saturday which you register through fair games attend a local event in your area the 14th of June brings a second sunday which is from 12 a.m. to 11.59 p.m. local time and during that time you're aiming to create one of the second sunday campaigns so whether that's scams missions have you want to do on there you can then be able to progress from that and then from mid-June we have the summer solstice 2026 operation slash campaign and that will span multiple weekends and we've got some more details so that as we get closer then finally the last Orion anomalous are on the 20th of June in Geneva Switzerland and Lima Peru during June we tend to have the pride XM activated on the first or around then so your scanner will look a bit prettier because you'll have all the red and yellow and green and pink and purple all those will be colours that appear on the map we may get a few more surprises during June I don't think we've got the courier files announced previously of what that was so may get a few more things coming up but that's probably it for June and then sometime in June we'll get the Q3 anomaly series and details for that I will mention here that the anomaly name for the Q3 series has been announced and it's sticking with the spatial theme being hollow obviously after the spacecrafts not after the animal crossing eagle character but it seems like they're sticking with the space theme for at least 2026 so I'm just wondering what they'll do next on their following Apollo and how that works into what the overall story seems to be going about to nemesis and Hank Johnson and possible law bits on there me yeah we could see what happens with that and what they actually do us we are waiting for Recon to come back and there's not had any updates on when that's happening I am still doing a bit of a way fair even though only countries to poke on go at the moment so it was interesting to see that the way fair team announced a new wayfinder rating system so if you used OPR before you used to the usual like poor good great kind of rating system on there 19 Tino posted on the 20th of May we're excited to launch the new wayfinder rating system version two and these changes are designed to offer not only great transparency also provide a clearer path of progressing or rating additionally this change to incorporate more of your activity within way fairer moving beyond the sole 30 review activity this new approach is built to recognise a broader range of your contributions approved nominations approved edits and agreements so the new wayfinder rating system consists of five levels rather than four so you go from novice which is zero to 99 apprentice which is 100 to 299 explorer which is 30699 expert which is seven hundred to one thousand four hundred ninety nine and champion which is the top level one thousand five hundred points 19 Tino mentions that your profile page now reflects change that have been applied retroactively and your new wayfinder rating companies all past accepted submissions including except nominations edits for those text locations 19 Tino also detailed to you progress in there so getting an approved nomination will get you plus three points getting approved edit will get you plus one two point five points and an agreement from reviewing will get you plus one point and you've got agreement points will we award for agreements and from nomination and edit reviews the interesting thing that I found is it comes with a few perks on there so within ingress we know that the amount of nominations used to be like 13 14 refreshing every so many days and then it changed to having 40 as the maximum one refreshed every day on there so you could use up 40 in a day then you get one back the next day one back the next day on there wayfarer's had a big shift now in what happens on there so if you are a novice you can only have up to ten nominations ten edits and ten well actually ten photo edits ten texts that's ten location edits and you accrue more nominations one a day if you're at the apprentice level do all those go up to 40 so as we know it on there and then getting one back a day on there one two views one the third level explorer jumps up it doubles it to 80 so could do 80 nominations in a or 80 photo edits and get those back two per day so you could use two nominations today and you get those back the next day expert doubles it again so you get 160 submissions photo edits text edits and location edits and that accrues back at five per day and then finally champion goes a bit crazy you get 320 submissions edits text edits and location edits and they come back at 10 per day so I can't imagine going out and nominating 320 portals in a day but it is technically possible to nominate 320 way spots in a day now through wayfarer if you're at the champion level and if you let your nominations build up so I imagine that's very handy for some people on there to be able to do that. Nantitian did note that more editions are composing more of your wafer activity will be coming soon as well as perks and lock as you progress through the new wayfinder rating some things are look forward to our submission distance increases and more to be announced at a later date so that would be great as well if ingress got that for extra submission distance based on how well you've been doing it is mixing annoyed the recon shutdown and then we get nothing more on there. A while back there was an investigation that an agent shared into some AI reviewing that happened on recon and that led to some enlightened agents probably not actual agents probably just bots that got suspended and they noted that those accounts are now all back online however they have had their recon badges removed So I see you see you. forcible removal on there, which I've not seen before. I know agents have requested some badges to be removed, but that seems like they've just basically put them back to zero on Recon to remove their badge entirely, whether they're locked out of it, whether it comes back. I don't know, I don't know how, banning users and Recon/Ingress would work on there, whether they had a system for that. But now that they've got that sorted, maybe a Recon revival is on the cards. For me, I don't think it's going to happen until at least September, because I'll close the Paradox game at the end of August, and then it'll be a case of seeing how things are working on there, seeing what ex-resources they've got spare to ingress. That was really well I've got for this week, folks, so thank you so much for listening. As I may do a pre-recorded episode next week, so it might just go over one subject, but I'll see how I feel after I get back how good I am. But for now, hope you have a great Sunday and happy hacking. Intel package transferred. For our latest yet to the outro, folks, I just want to share, I'm also doing a new podcast. So this is where I'm looking at a show for the first time, alongside someone that's watched lots of episodes of the show. The show itself is called Murder Drones, and the podcast you're looking for is Murder Drones Mayhem. As I said, something I've not done before, I'm kind of a rewatch show for one host, watch for the first time for me, where we discuss it on there, see what we think of it. The first episode should be dropping shortly after my podcast. So do look out for it and subscribe, and help out this new show on there. It will only be a certain number of episodes and like this show that goes on all the time, 'cause there's only a certain number of episodes to watch and review on there. So I appreciate you taking a bit of time just to listen to the first one that'll be great. Thank you very much all, happy hacking. Thank you, agent, for this is the latest episode of the Ingress Insights Podcast. If you'd like to support the podcast, you can do that in a couple of ways. So the easiest way to support the podcast is via Patreon. So you can find patreon.com/ingressinsights. And there's a free tier, a $1 a month tier, whether that is in your local currency, and a $5 a month tier. You check out what benefits those come with on the Patreon. One other thing you do that's totally free that helps out a lot is rating the podcast on your podcast service of choice. So where you can leave a rating or leave comments, that always helps increase visibility. And then finally you can follow us on various social media channels, you use most of them, but the best one to follow us on now is Blue Sky. So that's just ingressinsights.bsky.social. So that's your ingressinsights on Blue Sky, you'll find us on there. Finally, sharing the podcast with friends, family, other agents who want to listen and get updates or it helps out and helps get interviews with various ingress people or other game related people to then increase visibility more. So anything you can do from those is great. Thanks again for listening agents, have a great day.