Hi all,
We've been having a recurring problem where the SX100 ion pump apparently "sparks" from a metal flake or whisker and the vacuum logic interprets this momentary increase in current (and possibly small increase in pressure) as a vacuum failure, and therefore turns off the ion pump and high voltage and displays the "Turn Gun Valve To Position 3" message, thereby killing the run.
I guess one solution would be to clean the ion pump elements by banging on them with a hammer(!) and using a vacuum cleaner to collect the debris as we've done in the past, but the problem eventually returns. So we're curious if this "sparking" isn't as bad as the vacuum system thinks it is and that perhaps we could add a time constant in the vacuum logic to reduce the sensitivity to these events and average over a longer period of time (say 10 or 20 seconds), so the vacuum logic doesn't shutdown things so quickly if it's just a momentary event lasting only a second or two.
Does anyone have any idea how long these "spark" events last in the ion pump?
john
Quote from: Probeman on February 21, 2017, 12:29:01 PM
Hi all,
We've been having a recurring problem where the SX100 ion pump apparently "sparks" from a metal flake or whisker and the vacuum logic interprets this momentary increase in current (and possibly small increase in pressure) as a vacuum failure, and therefore turns off the ion pump and high voltage and displays the "Turn Gun Valve To Position 3" message, thereby killing the run.
I guess one solution would be to clean the ion pump elements by banging on them with a hammer(!) and using a vacuum cleaner to collect the debris as we've done in the past, but the problem eventually returns. So we're curious if this "sparking" isn't as bad as the vacuum system thinks it is and that perhaps we could add a time constant in the vacuum logic to reduce the sensitivity to these events and average over a longer period of time (say 10 or 20 seconds), so the vacuum logic doesn't shutdown things so quickly if it's just a momentary event lasting only a second or two.
Does anyone have any idea how long these "spark" events last in the ion pump?
john
A bit more data... we have eliminated the secondary gauge as a source of these "Turn Gun Valve To Position 3 " faults, so it appears the faults are from the ion pump in the gun.
Our ion pump elements are not more than 3 years old or so and our engineer says they look in good shape though there is some minor debris. His examination of the Cameca schematics show that the current "suppression time" for ion pump faults has a time constant of around 1 second or so, but that the ion pump faults which are knocking our filament off line can last as long as 5 seconds or so. He is collecting more data to improve that observation.
In the mean time I'm curious if other Cameca instruments with tungsten filaments also see these ion pump faults...
john
We have a similar issue, but it's a leak somewhere below the gun valve. Goes out every day or two, unless the valve is in position 1 (was fine for about 6 days). Very frustrating, and I couldn't say how long the disruptions last, but sometimes it's not long enough to shut off the vacuum, even if I see a momentary reading in the -3Pa range. Always fun to restart a run in the AM. It's like watching paint dry, if paint sounded like a giant metal spider.
I wish I wasn't on XP, so I could remotely monitor the vacuum status and drive in to restart if it went out, that would save so much time!
Quote from: neko on March 29, 2017, 12:46:09 PM
We have a similar issue, but it's a leak somewhere below the gun valve. Goes out every day or two, unless the valve is in position 1 (was fine for about 6 days). Very frustrating, and I couldn't say how long the disruptions last, but sometimes it's not long enough to shut off the vacuum, even if I see a momentary reading in the -3Pa range. Always fun to restart a run in the AM. It's like watching paint dry, if paint sounded like a giant metal spider.
I wish I wasn't on XP, so I could remotely monitor the vacuum status and drive in to restart if it went out, that would save so much time!
That is not a image I want to have in my head!
I should report that after our instrument engineer cleaned and scraped the ion pump elements we haven't had a single "Turn Gun Valve Position 3" problem yet and it's been about 3 weeks now! He mentioned that there were quite a large accumulation of tiny metal flakes clustered on the pump elements where the magnetic field was concentrated.
Note on remote access for Win XP. We use RealVNC for much of our remote access and it works great on all Windows and Mac platforms.
john
1. first thing is to clean the ion pump itself thoroughly. 2.check solid state relay inside vacuum chassis or better replace it 3.Also good idea to check ion pump HV connector contact.
This is not a replacement for properly cleaning the ion pump elements, but as a temporary solution one can just give some a good solid taps with a screwdriver handle to the side of the ion pump housing, which seems to prevent such glitches for a while. According to my instrument engineer, the theory is that small metal flakes and whiskers get broken off and fall to the bottom of the pump housing where they won't cause problems for a while.
Hi all
Yes we had this problem occurring all the time on our SX51... the solution was to do exactly what you are doing. Bang the crap out of the outside housing of the elements with the handle of a screwdriver for a while. This seemed to stop the problem occurring for a good number of months, until it happened again.
We had in the past when it got bad removed the elements, vacuumed housing, and got the elements sand blasted and recoated, but the people who did that for us went out of business.
Cheers
Quote from: BenjaminWade on October 08, 2017, 06:20:16 PM
Hi all
Yes we had this problem occurring all the time on our SX51... the solution was to do exactly what you are doing. Bang the crap out of the outside housing of the elements with the handle of a screwdriver for a while. This seemed to stop the problem occurring for a good number of months, until it happened again.
We had in the past when it got bad removed the elements, vacuumed housing, and got the elements sand blasted and recoated, but the people who did that for us went out of business.
Cheers
So the last time our engineer cleaned the ion pump elements we didn't see the "Turn Gun Valve Position 3" (ion pump glitch) message for about 6 months. But now it's returned with a vengeance. Does any one have any suggestions on cleaning the ion pump filaments (which seem othewise in fine shape) to avoid these "glitches"?
So can a leaky detector window be a problem on a Cameca (with a spectrometer separation window) even though the sample chamber vacuum is good?
Our SX100 almost definitely has a leaky window on SP5 and the ion pump tripping out seems to be associated with it. The vacuum in our WDS is pretty poor at the minute (or at least was last time I checked!), but the chamber vacuum is good (ours is a turbo pumped system). I also noticed some unusual behaviour on the bubbler of SP5.
Changing out the detector window was on the list of things to do before we all shut down, and it'll be at the top of the list when I get back (along with replace the blown filament).
Quote from: JonF on April 29, 2020, 07:36:55 AM
Our SX100 almost definitely has a leaky window on SP5 and the ion pump tripping out seems to be associated with it. The vacuum in our WDS is pretty poor at the minute (or at least was last time I checked!), but the chamber vacuum is good (ours is a turbo pumped system). I also noticed some unusual behaviour on the bubbler of SP5.
Changing out the detector window was on the list of things to do before we all shut down, and it'll be at the top of the list when I get back (along with replace the blown filament).
Breaking EPMA news: after a visual inspection our instrument engineer found a "pinhole" leak in one of our spectrometer/column separation windows, replaced it, pumped down, and our vacuum in the sample and gun have improved by an order of magnitude.
After 10 hours, so far so good.
Quote from: Probeman on April 29, 2020, 07:28:20 PM
Quote from: JonF on April 29, 2020, 07:36:55 AM
Our SX100 almost definitely has a leaky window on SP5 and the ion pump tripping out seems to be associated with it. The vacuum in our WDS is pretty poor at the minute (or at least was last time I checked!), but the chamber vacuum is good (ours is a turbo pumped system). I also noticed some unusual behaviour on the bubbler of SP5.
Changing out the detector window was on the list of things to do before we all shut down, and it'll be at the top of the list when I get back (along with replace the blown filament).
Breaking EPMA news: after a visual inspection our instrument engineer found a "pinhole" leak in one of our spectrometer/column separation windows, replaced it, pumped down, and our vacuum in the sample and gun have improved by an order of magnitude.
After 10 hours, so far so good.
Our gun and sample chamber vacuum hasn't been this good in years!
(https://smf.probesoftware.com/gallery/395_01_05_20_9_46_25.png)
If like Jon and I, you are seeing less than ideal vacuum readings in your Cameca and your ion pump keeps tripping, I think you ought to check your column separation windows for pin holes.
We also had Ar migrating into the ion pump causing it to arc from time to time. Was discussed in the sx50-users list around March 2019.
Somehow I missed that, but I can confirm that our ion pump is working great since we changed out the bad column separation window.
Here's a question: how does JEOL prevent "poisoning" of their ion pumps from P-10 argon if they don't have column separation windows? They must have very "tight" detector windows!
Update on ours: I found a bad detector window and having checked all the column separation windows, ended up replacing one thick film window that looked a bit blown (couldn't really see any holes though). Surprisingly, the thin film windows all looked pristine, so I'm not convinced any of the column windows were actually bad.
I stuck the detector window under a microscope and took a few pictures.
(https://smf.probesoftware.com/gallery/796_17_07_20_1_19_32.jpeg)
(https://smf.probesoftware.com/gallery/796_17_07_20_1_20_07.jpeg)
For scale, the window is approx 1 cm across the short dimension, the hole up to maybe 1 mm in length.
Typically, this window was from SP5 and was a pig to get out as we have the liquid nitrogen anti-contamination dewar fitted, severely reducing access.
Top tip: fill all the gaps in the table underneath with tin foil or blue roll! When the circlip holding the detector pops off, or when you drop it on refitting, it will fall in to the tin foil and not down in to the bottom of the instrument amongst the pipes and cabling. Saved my bacon, that did!
I'm waiting a while for the vacuum to recover before I restart the ion pump, but fingers crossed this has sorted out the ion pump tripping out!
I think this is good place to share my 2 cents.
This is going to be about SXFive FE, but don't discard this fast, particularly if you are planning to run LaB6 or CeB6 on your SX100 as that I think applies to that too (unless your ion pump is noble diode or StarCell type). So I see you guys had described here some serious leaks, and after fixing Ar leak from spectrometers the issues were fixed for your machines. If you are running with tungsten tip that is good approach and probably you won't need to look any further.
But for tips operating continuously a year and further you should be aware that there are also not-obvious Ar contamination sources. First of all very tiny part of Ar diffuses through separation window (despite tight seal). Secondly, there is always residue air on sample surface, pores, and air contains 1% of argon. This can look insignificant in short term, but is of key importance in long-term. These argon bits accumulate in the diode ion getter pump(s), but differently than other getterable gases, argon is temporary buried on cathode by sputtering process. The problem is that then pump gets critical mass of argon in the pump it is no more capable to manage to hold it all, and even worse - it looses ability to hold already collected argon... and so pump releases most of the argon in an "avalanche" fashion (thus You would see in the logged data huge spike).
I am describing here argon cycles, there can be other issues on SXFive FE like leaky metal gaskets (these can be checked or ruled out by changing room temperature as that is sensitive to that), and can produce similar patterns, which is hard to distinguish (devil in the details). It is good to understand how argon capture works, and how spiking can be overcome.
SXFive FE is equipped with two ion getter pumps (both simple Diode type). The big primary pump is situated close to the Schottky's emitter, and small (20 l) is situated below the primary. This small one buffers column vacuum in between chamber and gun containment section, and as discussed latter is first in the line to catch and suffer from argon accumulation. The small one starts to be Argon saturated after more-less 1 year (that is with excellent separation windows and also careful procedures limiting leakage of samples in the chamber; if spectrometer would leak argon, then pump would start to suffer much more sudden).
So whats going with that released argon from the pump? In closed system (sealed vacuum containment and single ion pump) that would go into "perpetual" well defined periodic spiking, as after argon release the same pump tries to catch that argon back. But after crossing some stability threshold it releases all captured argon again and repeats the process many times. On the microprobe it is a partially open system and part of released argon travels further in vacuum system where it is taken care. And so the affected pump with every cycle needs to deal with lesser amount of argon and duration in between spikes is increasing until finally it stops spiking. A small part of released argon escapes through column down to the chamber (if column-chamber valve is not closed) where it is taken out with turbo pumps, another small part is captured by other, in our case primary, pump. The affected pump keeps spiking despite lessened amount of argon as, differently to initially gradually accumulated argon, it needs to pump huge amount of argon at once. The newly formed argon burial structures are very unstable and so pump releases argon again very shortly after previous spike (minutes to hours to days). If part of argon can escape during spiking, then cycles will increase periodicity (longer time between spikes) as after some days/weeks/months there will be manageable amount of argon left and ion pump will cease spiking, but not forever, just for some longer time (months, half year...).
Looks good? Problem is self-solved? Not so fast. During smaller (secondary) pump release of argon, the primary ion pump is significantly being contaminated with argon with every spike of secondary pump, which is not good as primary pump being of simple Diode type is also susceptible for argon release. It is a larger pump and thus it can capture (temporary) much more argon... and so release much more argon. The safety vacuum threshold for Schottky emitter is set lax so that it would not get triggered with spikes originated in the secondary ion pump, but when primary pump gets argon saturated and releases all its argon - vacuum goes few orders over safety limit and trips the emitter.
If nothing is done with argon accumulation, then we experience that Primary pump releases argon in about 1.5 years from last baking. It actually did for us last week, that is why I am writing it here. I still have fresh mind after doing research on ion pump workings and workarounds. This is not first time it happen to us. We had similar "1.5 Year old" ion pump incident while running previous emitter. It then went worse than now as amount of argon released by primary pump was so large that secondary pump had gave up (that was 1.5 year ago, during Christmas-New Years eve brake, I was quite shocked to find such a mess getting back from holiday). This time secondary ion pump had survived, but I guess it is as I run it with manually set 5000V voltage, so it has much better response to pressure increase. But it was still enough to trip the field emission gun.
So I thought up some very simple procedure to get rid of Argon without baking the pumps, and I found it worked for our SXFive FE. It is based on 2 actually 3 assumptions:
1) If I want that no argon would be caught by any of the ion pumps, then both ion pumps should be off, as there is no possibility to isolate one from other.
2) Excess argon needs some open way to get out.
3) ion getter (Diode) pump needs energy to keep the Ar from escaping.
So with these in mind the procedure is clear:
(1) shut both ion pumps off
(2) while keeping column-chamber valve (EP 6) open.
(and obviously - the FEG needs to be shutdown before)
I had monitored the vacuum by separate vacuum gauge (IMG2(?)), which is installed near primary ion pump. The vacuum instantly worsened up.... but then as expected by my assumption, started to improve. As it had plateaued in a half hour, I had restarted ion pumps - vacuum got improved - no more spikes. Actually funnily, the primary pump did not wanted to restart, and controller showed "too low pressure" (too good vacuum) - seriously. But if waiting for a bit it gets correctly. It got back to ultimate vacuum in an hour.
BTW, emitter is today back on and running for SXFiveFE record, I have plans for it to be able to work stable (200 nA/1h within specification) up to 18000 hours... 5000h more to go. When FEG is run correctly it runs stable and also the vacuum baseline is really good. After argon riddance, IP1 (primary pump) shows 1.7E-7 Pa - that is with Shottky tip operational (!), before spikes (two weeks ago) it increased from 1.8E-7 to 2.0E-7 Pa. From previous experience, when FEG is run in nonsense default mode - the primary vacuum would raise to 2.8E-7 Pa in a few months from 1.7E-7 Pa achieved during tip installation. This demonstrates that actually any baking of ion pumps is not needed on FEG instrument (after installment) at all when everything is run using brains. (just for comparison - our ZEISS FEG SEM manuals highly recommend baking ion pumps once 1-3 months)
The long saga of our ion pump glitches (Turn Gun Valve to Position 3) may finally be over. And it appears that the problem was not due to a bad ion pump though we may have had some Ar contamination problems in the past.
Here is how our instrument engineer explained it to my lab manager:
QuoteHi Julie,
I may have stumbled upon the cause of the GV3 errors! I was going to pull a common mode choke off of the breaker assembly when I saw scorch marks around the wire contact holes on the bottom of the breaker. The wires crimped there were heavily oxidized, which would make for a very poor connection at this point. Since these wires feed the Physics Subsystem, that subsystem was likely running in an undervoltage condition, leading to instability there.
Also, current cutting in and out because of the bad connections would lead to noise transients that triggered the Ion Pump and Penning safety circuits. I've noticed that such transients bleed right into the power supply to those safety circuits.
Tomorrow AM I'll clean up the breakers contacts and trim back the wires to fresh unoxidized copper. We'll see if this is the cause.
Cameca is working on getting us the proper part this time. I reckon we should allow them to send it to us since we're not 100% sure the breaker problem is the cause.
Cheers,
Steve
He then followed that up with this further explanation to me:
QuoteHi John,
Read the email below to see what was found as the cause of the GV3 error.
Normally our fix for the error was to bang ion the ion pump to knock out arcing ash, or replace a column separation window to get Chamber and Gun pressures low enough that system noise would not trigger the Vac CPUs TTL fault monitor inputs. When this new version of the GV3 error crept in, nothing could prevent it except for lying (via a Lie Generator Box I had to build) to the pressure measurement system and the Vac CPU's TTL fault monitor inputs.
Regarding the note to Julie below, I wanted that common mode choke to check if it could reduce the noise bleed through that the +/-15V power supply exhibits to a ridiculous degree on the test bench. That's when I discovered the scorched connections on the circuit breaker. Bark up the right tree and you'll eventually spot the prey! Note that it turns out that the +/-15V supply would not have benefited from that common mode choke because that supply is on the exit side of the choke and would thus be exposed to full amplitude noise from the bad connections on the very same circuit node. Also note that screws of the scorched terminals of the breaker were under nearly zero torque (i.e. they took almost no torque to loosen with a screw driver). All owners of SX100's would do well to re-tighten these screws. If they find scorching and overheated wires, they should wire brush the breakers wire crimps and replace the overheated wires (same 10AWG stranded hookup wire or larger).
Reusing overheated wire is not a good idea as thermal runaway will likely recur.
Cheers,
Steve
Although the frequency of the vacuum glitches described in this topic have decreased since they began, we are still seeing them occasionally.
I've asked our instrument engineer to summarize the situation in case any one has seen similar issues on their Cameca instrument and/or any suggestions on how to further mitigate these glitches:
DESCRIPTION OF THE PROBLEM (GV3 Error):UO's Cameca SX100 microprobe suffers from vacuum shut downs that occur whether or not the beam is on. There is some evidence that the problem occurs more frequently when the beam is on, which is understandable since the extra electrical noise background is higher with high voltage on. See further details below. These shutdowns can interrupt a long analysis run, or a mapping run. We call these shutdowns GV3 errors because the user interface error simply says Please Set Gun Valve to Position 3. When we comply with this dialog request, the vacuum system begins a sequence which restores full diffusion pump and Ion pump operation.
CAMECA SX100 Vacuum Pressure Measurement System:There are six vacuum pressure gauges feeding vacuum pressure readings to the vacuum control board. Four of these are thermocouple gauges for measuring rough vacuum. One is a penning gauge, and the remaining is the ion pump for the electron gun chamber. Each of these six gauges provide an analog voltage that corresponds to the measured pressure and these values are displayed in the user interface. These analog voltages are also monitored by safety comparator circuits that will detect when the readings of any given gauge have risen above a safe operating level. The logic level output of these comparators feed the vacuum control processor and will cause a vacuum system shutdown when they go active high (+5V). The vacuum pressure readings from the analog signals can also cause a shutdown when they read too high, even if the comparator outputs have been defeated. So there are redundant means determining if a given gauge is experiencing out of bounds pressure: Analog pressure readings that Cameca software will respond to, and Safety comparators (who's inputs are those Analog voltages). If you were to defeat a comparator output (force it Low), a high Analog voltage caused by a truly high pressure level will still cause a shutdown, although with an unknown latency.
Determining which gauge triggered the vacuum shutdown:To our knowledge, there is no logfile that captures these errors and discloses which of the six gauges had seen the pressure excursion that led to the shutdown. We have installed a custom state logger that can capture these comparator outputs and we now have knowledge of which gauge experienced a pressure burst or noise. Only a logger such as this with a time resolution of 10mS or faster can reveal what's happening because once a comparator has been tripped, several more will trip due to the vacuum system shut down. This solution to the Who Dunnit problem works, but it does require importing the logger contents into a spreadsheet and producing signal vs time plots.
We also have defeated these comparator logic level outputs during tests to see if we could determine which thermocouple gauge was source of GV3 errors, and this did work. It turns out that our Varian TC controller has some noise issues since we have all new TC gauges, but still have GV3 errors related to them.
MEASURES TO REDUCE GV3 ERROR RATE:
- Replace the gun ion pump elements (and clean out sputter ash) if the gun pressure is consistently 5x10e-5 Pa or higher
- Replace column separation windows with pinhole leaks that lead to Col. pressure (penning) readings consistently 5x10e-4 or higher
- Clean any debris on gate valve O-ring that leads to Col. pressure (penning) readings consistently 5x10e-4 or higher
- Replace thermocouple gauges that have gone noisy (readings on the Varian controller can reveal this noise)
The 4 measures above will keep the gauge analog voltages (which represents pressure) further away from the safety comparator trip thresholds, thus preventing nuisance GV3 errors when system noise riding on top of the analog readings crosses the thresholds.
ADDITIONAL MEASURES (To no avail):We have also doubled or tripled the value of capacitor C1 in both penning and ion pump power supplies in order to suppress transients due to arcs. These changes had no effect on GV3 errors.
GV3 ERRORS ARE LESS FREQUENT, BUT STILL OCCUR:We still get the GV3 errors, though much less frequently due to the measures taken so far. The design of the vacuum system seems sound when it comes to timely response to real vacuum pressure excursions. The problem is that this same safety circuit can respond to system noise and cause nuisance shutdowns. Data logger runs (not the State Logger mentioned above) have not revealed any real pressure burst when one of these GV3 shutdowns has occurred. This leads us to believe system noise is responsible for the tripped comparator thresholds.
This state of affairs has us considering designing a comparator signal vetting circuit which will reject short duration events from the comparators. If the event duration exceeds perhaps 10mS, or manifests as a burst of short pulses, the signal will be immediately passed on to the vacuum processor.
For further safety, we'll use combinatorial logic to pass the fault immediately if two or more gauges is seeing an event.